For site owners

About CrawlStatBot.

If you saw CrawlStatBot in your logs, someone ran the CrawlStat desktop app against your site, usually to audit their own.

The user agent looks like this:

CrawlStatBot/1.0 (+https://crawlstat.com/bot)

How it behaves

  • It follows robots.txt by default, both the Disallow rules and Crawl-delay.
  • It paces requests per host, with 5 simultaneous requests by default.
  • It runs on the user’s own computer. There is no CrawlStat server crawling the web, and no data is collected by us.
  • It is open source, so you can read exactly how it behaves.

Blocking or slowing it

Add a group for CrawlStatBot to your robots.txt:

User-agent: CrawlStatBot
Disallow: /

# or, to slow it down instead:
User-agent: CrawlStatBot
Crawl-delay: 2

Questions or a problem? Get in touch. For more, read The crawler and robots.txt.