Competitor Intelligence
Compare your site with up to five competitors and see what they cover that you don't, what to write next, and how ready each site is for AI search.
What it is for
Competitor Intelligence answers a simple question: what do my competitors cover that I don't?
You give it your site and up to five competitors. CrawlStat crawls them all side by side, works out what each site's pages are about, and shows you:
- the topics, questions and phrases they cover and you are missing, or cover much more thinly
- which of your own pages compete with each other
- a content plan, sorted by where a visitor is in their journey
- how ready each site is for AI search
Everything runs on your computer. Nothing about your site or your competitors is uploaded.
Start an analysis
Setup has three steps:
- Your site. Type the address of your own site.
- Competitors. Add up to five competitor sites.
- How deep. Choose how many pages to crawl on each site, then click Start analysis.
| Depth | Pages per site | About how long | Good for |
|---|---|---|---|
| Quick | 200 | 1 minute | The main pages of each site. A first look. |
| Standard | 500 | 3 minutes | Enough to see each site's topics. This is the one we recommend. |
| Thorough | 2,000 | 9 minutes | Large sites, looked at in depth. |
Times are rough. Slow sites take longer.
All the sites are crawled at the same time, and politely: two requests at a time per site, a short pause between them, and robots.txt is respected on every site. See The crawler and robots.txt.
The tabs
When the analysis is ready, it has six tabs: Overview, Competitors, Gaps, Topics, Content plan and Page check. Overview sums up the whole analysis. The others are below.
Competitors
Each competitor gets a card with a label, such as Direct competitor, and these figures:
| Figure | What it means |
|---|---|
| Pages | How many of its pages were crawled. |
| Topics | How many topics its pages fall into. |
| Unique topics | Topics this site covers that the others don't. |
| Phrases covered | How many search phrases its pages cover. |
| Overlap with you | How much of what it covers you cover too, as a percentage. |
| Authority (est.) | A rough guess at how strong the site is. |
| Freshness | How many pages were updated recently, for example "151 in 90 days". |
| AI readiness | A score out of 100. See Page check. |
| Opportunities | How many gaps this site shows you. |
The Funnel mix bar shows how its pages split between Learn, Compare and Buy. Gaps against this site opens the Gaps tab for just that competitor. You can also recrawl, configure or remove each site from its card.
Gaps
Gaps lists the things competitors cover that you are missing, or cover much more thinly. Click one to see the evidence: which competitor pages cover it, and how.
The sub tabs split them up:
- To act on: the ones most worth your time.
- Missing: competitors cover it, and you don't at all.
- Weaker than them: you cover it, but much more thinly.
- Covered in other words: you do cover it, just with different wording.
- Your pages competing: two or more of your own pages chase the same thing.
You can filter by kind (Keywords, Questions, Long tail, Names, Topics, Content types), by stage (Learn, Compare, Buy, Unclear), by confidence (High, Medium, Low), by competitor and by a minimum score.
The table shows a Score (marked "est."), the Opportunity, its Stage, the Competitors that cover it (one coloured dot per competitor) and its Status.
Topics
A topic is a group of pages, from every site, that are about the same thing. CrawlStat finds them by reading the meaning of each page and grouping pages that are close.
There are two views:
- Map: each circle is a topic. A bigger circle means more pages. Lines join related topics, and a dashed line means your site doesn't link those topics together yet. Scroll to zoom and drag to move.
- List: the same topics as a list, with how many of your pages are in each.
The colour shows where you stand: You are missing it, You are weaker, You cover it or Only you cover it. You can filter by stage and by site, turn Show connections on or off, and use Reset my edits to undo your changes.
Content plan
The content plan has two parts: Opportunities and Briefs. Opportunities are sorted into three columns, by where a visitor is in their journey:
| Stage | What kind of content | How you measure it |
|---|---|---|
| Learn (TOFU) | Articles that teach, explainers, guides for beginners, definitions | Reach |
| Compare (MOFU) | Comparisons, case studies, how to guides, templates | Sign ups and leads |
| Buy (BOFU) | Pricing, alternatives, product and service pages, ROI calculators | Sales |
Each opportunity comes with a suggestion, for example: Answer "When do I need to hire an electrician?" as an H2 with a direct 40 to 60 word answer.
Click Create brief to turn an opportunity into a brief: an outline, ready to give to a writer, built from what competitors cover and what they leave out. Your briefs are kept under Briefs.
Page check
Page check opens with the AI search readiness scorecard. It compares you with each competitor, check by check, as the share of pages that pass:
- average score
- direct answer near the top
- concise answer passages
- explicit definitions
- factual statements that are easy to pull out
- well structured headings
- structured questions and answers
- clear introductory summary
- author and organisation
- expert attribution
- supporting references
- original research or first hand experience
- dated and recently updated
- FAQ markup
- page type markup
The scorecard measures what a page's markup shows an answer engine. It doesn't tell you whether any AI system will quote the page.
You can also check any one of your own pages here, and get drafted changes that you can approve.
What the scores mean
Every score in Competitor Intelligence is an estimate, and the app labels them "est.". They come from your crawl and your competitors' crawls, not from search engine data or your analytics. Use them to decide what to look at first, not as a promise of traffic.
Local AI (optional)
Competitor Intelligence can use Ollama on your own machine in two ways:
- Meaning matching. A small embedding model matches pages that say the same thing in different words. Install it with
ollama pull all-minilm. - Topic names and brief drafts. A local chat model gives topics clear names and helps draft briefs.
Both are optional, and both run on your computer. See Local AI.
Monthly rechecks
In setup, turn on Recheck competitors every month. CrawlStat then crawls your competitors again each month while CrawlStat is open, and shows you what changed. You can change this for each site later.
To recheck now, click Recheck at the top of the screen. You can also run rechecks from the command line, for example on a schedule:
crawlstat-cli intel recheck
For AI agents
The same tools are open to AI agents through an MCP server. Start it with:
crawlstat-cli intel mcp
It talks over stdio, so the agent that starts it is the only thing that can use it.
Privacy
Competitor Intelligence runs on your machine, like the rest of CrawlStat. Your competitors' sites are crawled from your computer, and the results stay on your computer. The local AI models run on your computer too. See Privacy and your data.