AhrefsBot

What is AhrefsBot?

AhrefsBot is a Web Crawler that powers the 12 trillion link database for Ahrefs online marketing toolset. It constantly crawls web to fill our database with new links and check the status of the previously found ones to provide the most comprehensive and up-to-the-minute data to our users.

Link data collected by Ahrefs Bot from the web is used by thousands of digital marketers around the world to plan, execute, and monitor their online marketing campaigns.

This data has immense value for SEO community as it helps marketing professionals to better understand the fundamental algorithms of the world's largest search engines so that they could optimize websites accordingly. Every 24 hours our crawler visits over 6 billion web pages and updates Ahrefs’ index every 15-30 minutes.

According to a recent third-party study of “good bots”, AhrefsBot is the second most active crawler after Googlebot.

What is AhrefsBot doing on your website?

AhrefsBot is crawling your website making notes of outbound links and adding them to our database. It will periodically re-crawl your website to check the current status of previously found links.

Our crawler does not collect or store any other information about your website. It does not trigger ads on your website (if any) and won’t add numbers to your Google Analytics traffic.

Does it respect robots.txt file?

Yes. Absolutely.

We strictly respect robots.txt, both disallow and allow rules.

How to control AhrefsBot on your website?

As mentioned above, AhrefsBot strictly follows the robots.txt file on your website. So you can fully control it on your website if you need.

To change the frequency of AhrefsBot visiting your site, you can specify the minimum acceptable delay between two consecutive requests from our bot in your robots.txt file:

Where Crawl-Delay value is time in seconds.

If for some reason you want to prevent AhrefsBot from visiting your site, put the two following lines into the robots.txt file on your server:

Please note that AhrefsBot may need some time to pick the changes in your robots.txt file. This will be made prior to each next scheduled crawl.

Please also note that if your robots.txt contains errors and AhrefsBot won’t be able to recognize your commands it will continue crawling your website the way it did before.

Read more about robots.txt at www.robotstxt.org.

If you think that AhrefsBot is someway misbehaving on your website or if you have any questions about it, please don’t hesitate to contact our support team [email protected]