• Home
  • Get help
  • Ask a question
Last post 23 min ago
Posts last week 110
Average response time last week 29 min
All time posts 67394
All time tickets 10423
All time avg. posts per day 21

Helpdesk is open from Monday through Friday CET

Please create an (free) account to post any question in the support area.
Please check the development versions area. Look at the changelog, maybe your specific problem has been resolved already!
All tickets are private and they cannot be viewed by anyone. We have made public only a few tickets that we found helpful, after removing private information from them.

#8066 – Ahres site adult can not crawl my site

Posted in ‘4SEO’
This is a public ticket. Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.
Sunday, 08 August 2021 05:19 UTC
kiro789

Hello,

I used robots.txt optimize for SEO button , but now Ahres can not crawl my site. What should I do to let Ahres site adult crawl my site.

Regards!

Sunday, 08 August 2021 05:43 UTC
kiro789

And Semrush can not crawl the site. Hier is the error log.

Monday, 09 August 2021 06:58 UTC
wb_weeblr

Hi

The Semruish shows a communitcation error with your site, nothing related to the robots.txt. Your current robots.txt is perfectly fine and does not block anything. 

Actually, changes made to robots.txt by 4SEO are opening folders, it does not restrict access more than Joomla, it allows more access.

But again, the error shown by Semrush is a communication error, nothing related to robots.txt.

Note that I tried that curl command today and I get a different error - but still an error:

Considering your site works right now, my best bet is that you have a security system in place preventing access by bots. It can be either in your Joomla site or at server lever so that's where I would look next: first disable any security-related extension running on your site and test again. Then contact your hosting company describing the problem and providing the Semrush error log. Semrush provides their bot IP address so you might have to white list them.

Best regards

Yannick Gaultier

weeblr.com / @weeblr

 

 
Wednesday, 11 August 2021 06:58 UTC
kiro789

Hello,

Ok, Ticket resolved. I asked the hosting to add IPs form semrush to whitelist.

Thanks!

Wednesday, 11 August 2021 07:03 UTC
wb_weeblr

Hi

Glad it's sorted out! Closing this ticket now, feel free to open a new one as needed. If you do so, please mention this ticket number in the new one.

Best regards

Yannick Gaultier

weeblr.com / @weeblr

 

 
This ticket is closed, therefore read-only. You can no longer reply to it. If you need to provide more information, please open a new ticket and mention this ticket's number.