Privacy and your data
What CrawlStat stores on your computer, the little it ever sends over the network, and how to back up or delete your data.
The short version
CrawlStat is a local app. Your crawls run on your computer, are stored on your computer, and stay there. There's no CrawlStat account, no cloud sync, and no analytics or tracking inside the app.
The only things that leave your machine are:
- The requests CrawlStat makes to the sites you crawl, which is the point of a crawler.
- Update checks, which ask crawlstat.com for the latest version number. They contain nothing about you or your crawls.
The rest of this page explains each of these in detail.
What's stored on your computer
| What | Where | Contains |
|---|---|---|
| Crawls | One .db file per crawl in the projects folder | Every URL crawled, status codes, response headers, titles, descriptions, headings, links, images, structured data, the crawl's settings and log, and audit results. Page source too, if you turned on Keep raw HTML. |
| Settings | settings.json | Your crawl defaults, theme and window preferences. |
| Recent crawls | recent.json | The paths of your 20 most recent crawls. |
| Exports | Your Downloads folder, by default | The CSV and Markdown files and client reports you've exported. |
The projects folder, settings.json and recent.json live in CrawlStat's application data folder. See Installing for the path on each platform.
By default, CrawlStat stores a fixed set of response headers rather than all of them: content-type, content-length, content-encoding, content-language, cache-control, etag, expires, last-modified, location, link, server, vary, x-robots-tag, strict-transport-security, x-frame-options and content-security-policy. It doesn't store cookies.
What's sent over the network
To the sites you crawl
CrawlStat sends ordinary HTTP requests to the sites you point it at, identifying itself as CrawlStatBot/1.0 (+https://crawlstat.com/bot) unless you change the user agent. If Check external links is on, it also requests each external URL your site links to, once, to see whether it works. If you turn on Render JavaScript, pages are also loaded in a browser on your machine, with a fresh, empty profile each time, so none of your own cookies or logins are used. See The crawler and robots.txt.
Competitor Intelligence works the same way. Your competitors' sites are crawled from your computer, politely (two requests at a time per site, with a short pause between them, and robots.txt respected), and everything it finds and works out stays on your machine. The optional local AI runs on your machine too. The MCP server for AI agents only talks to the agent that starts it on your computer; if that agent uses a cloud model, what it does with the results is up to that agent.
Client reports are made on your machine too. Printing one to PDF uses your own browser, offline, and the report is just a file in your export folder until you choose to send it.
No licence checks
CrawlStat is free and open source, so there's no licence, no key and no account. The app never contacts a licence service.
Update checks
Information about the latest release (its version number, release notes and download links) is published as a small file on crawlstat.com. Fetching it sends nothing about you or your crawls beyond what any web request carries, such as your IP address.
Nothing else
The app has no telemetry, usage analytics or crash reporting. The code is open source, so anyone can check this. The Settings screen puts it plainly: crawls run entirely on this machine.
Backing up your data
Each crawl is a single file, so backing up is simple:
- Copy individual
.dbfiles from theprojectsfolder to keep particular crawls. - Back up the whole application data folder to keep everything, including settings and your recent list.
Copy crawl files while CrawlStat isn't using them. There's no licence or key to move, so a new computer only needs CrawlStat and your files.
Deleting your data
- One crawl: on the Projects screen, use ⋯ → Delete crawl file…. This deletes the file and its
-waland-shmcompanions. - Exports: delete the CSV files from your Downloads folder as you would any other file.
- Everything: quit CrawlStat, then delete the application data folder.
There's nothing to delete on our side for your crawl data, because we never had it.