CrawlStat is now free and open source
Every feature is now free for everyone, with no account, no key and no limits. The code is on GitHub under the MIT licence. Here's why we did it, what's new, and how you can help.
Today CrawlStat becomes free and open source, under the MIT licence. The code for the crawl engine and the desktop app is on GitHub. You can read it, build it, change it and share it.
Everything is free now
There is no paid plan anymore. Every feature is included for everyone:
- unlimited URLs in every crawl,
- every audit, on every page,
- UX Testing,
- crawl comparison,
- progress in client reports,
- and the new Competitor Intelligence.
There's no account to make, no key to paste, nothing to activate and no limit on how many URLs a crawl can fetch. You download it and use it.
Why we did it
A tool that reads your sites should be one you can read too. A crawler reads your sites, and often your clients' sites too. Now you can check exactly what it fetches, what it stores and where it keeps it. (On your disk, and nowhere else.)
Trust is easier to give when you can check. When you point a tool at a client's site, "trust us" isn't a great answer. "Here's the code" is a better one.
It runs on your machine anyway. CrawlStat never needed our servers to crawl. The only part that ever talked to us was the licence check, and the official build doesn't have one anymore.
More people can make it better. We're a small team. SEOs find bugs we'd never think of, on sites we'd never see. Developers can fix them. Open source makes it easy for both to help.
What's new: Competitor Intelligence
Competitor Intelligence is the biggest new feature since launch. It compares your site with up to five competitors and shows you what they cover that you don't.
You add your site and your competitors, and CrawlStat crawls them side by side. It stays polite: two requests at a time per site, and robots.txt is respected. You choose how deep to go. Quick reads 200 pages per site (about a minute), Standard reads 500 (about three minutes) and Thorough reads 2,000 (about nine minutes).
The screenshots below show a local electrician compared with four competitors. We blurred the real site names.
The Competitors tab puts each site next to yours: its pages, topics, the topics only it covers, how much it overlaps with you, an authority estimate, freshness, AI readiness and the opportunities it shows.
The Gaps tab lists what competitors cover that you're missing, or cover only thinly. Each gap comes with the evidence, so you can see why it's there. You can filter by keywords, questions, long tail phrases, names, topics and content types, and by stage: Learn, Compare or Buy.
The Topics tab groups pages from every site by meaning. You can see the topics as a map or as a list, coloured by where you stand: you're missing it, you're weaker, you cover it, or only you cover it.
The Content plan tab sorts the opportunities into three columns: Learn, Compare and Buy. Each one has a Create brief button. It writes an outline that's ready to hand to a writer, built from what competitors cover and what they leave out.
The Page check tab is a scorecard for AI search readiness, for your pages and theirs. It looks for direct answers near the top, clear headings, FAQ markup, an author, dates and more. It measures what the markup shows. It can't tell you whether an AI will actually cite your page.
Local AI, if you want it
If you run Ollama, CrawlStat can use it on your machine. A small embedding model (all-minilm) matches gaps by meaning, so "cost breakdown" and "pricing" aren't counted as a gap just because the words differ. A local chat model can name the topics and draft the briefs. Both are optional.
A few more things
- Competitors can be rechecked every month, automatically.
- The same tools are open to AI agents through an MCP server:
crawlstat-cli intel mcp. - All of it runs on your machine.
- Scores are estimates, and the app labels them that way.
The Competitor Intelligence guide walks through every tab.
How you can help
Here's where help counts most right now.
Test it on your own sites
Crawl the sites you know best and tell us when something looks wrong. We especially need testers on macOS and Windows. The Linux installers are out, but the macOS and Windows installers aren't published yet. For now, you can build CrawlStat from source on those platforms. The README on GitHub explains how.
Report bugs
Open an issue on GitHub. It helps a lot if you include:
- the CrawlStat version,
- your platform (macOS, Windows or Linux, and which version),
- the smallest site that shows the problem,
- what you expected, and what happened instead.
Please only crawl sites you own or have permission to test.
Write code
The crawl engine is written in Rust and lives in engine/. The desktop app is written in Flutter and lives in app/. The logic lives in Rust, and the app shows the results. Before you open a pull request, please run:
cargo test
cargo clippy --all-targets
cd app && flutter analyze && flutter test
The contributing guide has the rest.
Improve the docs and this website
If a help page confused you, it will confuse someone else too. Fixes and ideas for the docs and for this website are just as welcome as code.
Star the repo
A star on GitHub takes one click, and it helps more people find CrawlStat.
Get started
- Read the code, and star it, on GitHub.
- See every way to help on the contribute page.
- Download CrawlStat and point it at your own site.
Thank you for crawling with us.