Notes from the crawl.
How CrawlStat is built, and how to fix what it finds.
CrawlStat is now free and open source
Every feature is now free for everyone, with no account, no key and no limits. The code is on GitHub under the MIT licence. Here's why we did it, what's new, and how you can help.
Read the postClient reports, JavaScript rendering and a new way around a crawl
CrawlStat now turns a crawl into a branded PDF for your client, renders sites built with JavaScript in the browser you already have, and gives every crawl its own page. Here's what changed.
ReadIntroducing CrawlStat
A desktop SEO crawler that is fast, light on memory and pleasant to spend an afternoon in. It runs on your machine, and it is free and open source with every feature included.
ReadThe database is the frontier: how CrawlStat keeps memory flat
CrawlStat has no crawl queue in memory. The crawl file is the queue, the list of seen URLs and the results, which is why 50,000 URLs fit in 128 MiB and doubling a crawl barely moves peak memory.
ReadRedirect chains: the chain is the finding
How redirect chains and loops build up, why CrawlStat follows every redirect by hand instead of letting the HTTP client do it, and a practical routine for collapsing them.
ReadLicensing without a database
An engineering story. How the licence code in CrawlStat checks a signed token offline, so a crawl never waits on a server, and the delay we accepted on purpose to get there.
ReadStart crawling in a minute.
Download CrawlStat, paste a URL, press Enter. Free and open source, with every feature. No account, no key, no limits.