Hello,
I used robots.txt optimize for SEO button , but now Ahres can not crawl my site. What should I do to let Ahres site adult crawl my site.
Regards!
Last post
26 min ago
Posts last week
93
Average response time last week
59 min
All time posts
70523
All time tickets
10886
All time avg. posts per day
20
Helpdesk is open from Monday through Friday CET
Please create an (free) account to post any question in the support area.
Please check the development versions area. Look at the changelog, maybe your specific problem has been resolved already!
All tickets are private and they cannot be viewed by anyone. We have made public only a few tickets that we found helpful, after removing private information from them.
#8066 – Ahres site adult can not crawl my site
Posted in ‘4SEO’
This is a public ticket. Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.
kiro789
kiro789
And Semrush can not crawl the site. Hier is the error log.
wb_weeblr
Hi
The Semruish shows a communitcation error with your site, nothing related to the robots.txt. Your current robots.txt is perfectly fine and does not block anything.
Actually, changes made to robots.txt by 4SEO are opening folders, it does not restrict access more than Joomla, it allows more access.
But again, the error shown by Semrush is a communication error, nothing related to robots.txt.
Note that I tried that curl command today and I get a different error - but still an error:
Considering your site works right now, my best bet is that you have a security system in place preventing access by bots. It can be either in your Joomla site or at server lever so that's where I would look next: first disable any security-related extension running on your site and test again. Then contact your hosting company describing the problem and providing the Semrush error log. Semrush provides their bot IP address so you might have to white list them.
Best regards
Yannick Gaultier
weeblr.com / @weeblr
kiro789
Hello,
Ok, Ticket resolved. I asked the hosting to add IPs form semrush to whitelist.
Thanks!
wb_weeblr
Hi
Glad it's sorted out! Closing this ticket now, feel free to open a new one as needed. If you do so, please mention this ticket number in the new one.
Best regards
Yannick Gaultier
weeblr.com / @weeblr
This ticket is closed, therefore read-only. You can no longer reply to it. If you need to provide more information, please open a new ticket and mention this ticket's number.