[{"data":1,"prerenderedAt":389},["ShallowReactive",2],{"blog-\u002Fblog\u002Fflat-memory-crawling":3,"blog-more-\u002Fblog\u002Fflat-memory-crawling":362},{"id":4,"title":5,"author":6,"body":7,"cover":350,"date":351,"description":352,"extension":353,"meta":354,"navigation":355,"path":356,"readingTime":357,"seo":358,"stem":359,"tag":360,"__hash__":361},"blog\u002Fblog\u002Fflat-memory-crawling.md","The database is the frontier: how CrawlStat keeps memory flat","The CrawlStat team",{"type":8,"value":9,"toc":339},"minimark",[10,14,22,25,28,33,36,43,46,58,62,65,77,80,84,87,94,97,131,134,138,141,277,283,290,293,300,304,307,310,314,317,320,324],[11,12,13],"p",{},"Most crawlers have a memory problem, and it's built into how they're designed.",[11,15,16,17,21],{},"A crawler has to keep track of three things: the URLs it still has to visit (the ",[18,19,20],"em",{},"frontier","), the URLs it has already seen (so it doesn't fetch them twice), and everything it learned from the pages it has fetched. The obvious way to build one keeps all three in RAM. A queue, a set, and a growing pile of page records.",[11,23,24],{},"That works fine until it doesn't. Every one of those structures grows with the size of the crawl. The link graph grows even faster, because a page has many links. Crawl a big enough site and you're watching the memory graph, tuning heap sizes, and wondering whether the crawl will finish before the machine starts swapping.",[11,26,27],{},"We wanted crawl size to depend on time and disk space, not on RAM. Here's how CrawlStat does that.",[29,30,32],"h2",{"id":31},"there-is-no-queue","There is no queue",[11,34,35],{},"CrawlStat has no queue in memory, and nothing in memory that lists the URLs it has already seen. Both of those jobs belong to the crawl file on your disk.",[11,37,38,39,42],{},"Every URL the crawler discovers is written to that file with a ",[18,40,41],{},"dedup key",", and the file refuses to store the same key twice. When the crawler finds links on a page, it simply writes them all down and lets the file reject the ones it already knows about. Nothing has to be checked in memory first.",[11,44,45],{},"Taking work off the frontier is just as simple. The crawl loop claims a batch of waiting URLs, shallowest first, and marks them as in progress in the same step. Ordering by depth makes the crawl breadth first, and a stable tiebreaker makes the order repeatable for a given site. The lookup stays fast however large the crawl gets.",[11,47,48,49,53,54,57],{},"The dedup key is deliberately not the URL itself. The URL we fetch only gets changes that are safe and don't alter meaning: a lowercased host, no fragment, no default port, resolved dot segments. Your policy choices (trailing slashes, tracking parameters, ",[50,51,52],"code",{},"index.html",") only affect the ",[18,55,56],{},"key",". So the first version of a URL the crawler sees is the version it fetches and shows you, and later variants of the same page are skipped.",[29,59,61],{"id":60},"one-writer-one-save-per-batch","One writer, one save per batch",[11,63,64],{},"Pages are fetched concurrently, but all writing happens in one place. The fetchers hand their results to a single writer, which saves them in batches.",[11,66,67,68,72,73,76],{},"Each batch is saved ",[69,70,71],"strong",{},"all at once",". It holds the finished pages ",[18,74,75],{},"and"," the URLs discovered on them, so either both are saved or neither is. A crash can never leave a page marked done while the links it found are missing. That's what makes a crawl resumable: open it from the Projects screen and press Resume, and it carries on exactly where it stopped.",[11,78,79],{},"You get crash safety without any recovery step, because the state you'd need to recover was never in memory in the first place.",[29,81,83],{"id":82},"why-memory-plateaus","Why memory plateaus",[11,85,86],{},"If the crawl state lives on disk, what's using RAM? Mostly a cache of fixed size that keeps the busiest part of the frontier close at hand. We cap it at 64 MiB.",[11,88,89,90,93],{},"That cache is a ",[69,91,92],{},"fixed"," cost. A crawl fills it, and from then on it doesn't grow, however many URLs you add. That's the plateau you see in the benchmarks.",[11,95,96],{},"A few other decisions protect that property:",[98,99,100,107,113,119,125],"ul",{},[101,102,103,106],"li",{},[69,104,105],{},"Big sorts happen on disk."," Building lookups over a few hundred thousand rows involves a lot of sorting. Doing that sorting in RAM is exactly the growth we're trying to avoid.",[101,108,109,112],{},[69,110,111],{},"Analysis lookups are built at the end."," The crawl only builds what it needs while it runs. Everything used for analysis gets built once the crawl finishes, so it doesn't slow down hundreds of thousands of writes.",[101,114,115,118],{},[69,116,117],{},"Links point at the target's dedup key."," A page's links can be saved in one go, without first looking up targets that may not have been discovered yet.",[101,120,121,124],{},[69,122,123],{},"Link score is computed on disk."," CrawlStat's link score is PageRank (damping 0.85, rescaled so the strongest page is 100). It runs in passes over the saved links, not over a graph held in memory. The link list grows with the crawl, so holding it in memory would break everything above.",[101,126,127,130],{},[69,128,129],{},"Reading never blocks writing."," The app can show you results while the crawler is still saving new ones.",[11,132,133],{},"The interface follows the same rule. The results table fetches 200 rows at a time as you scroll and keeps at most 20 pages in memory. While a crawl runs, the live view keeps only the last 300 URLs, since every one of them is already on disk.",[29,135,137],{"id":136},"the-numbers","The numbers",[11,139,140],{},"We measured on an Apple Silicon laptop against a local test site served over loopback. So these numbers measure the engine, not the network:",[142,143,144,166],"table",{},[145,146,147],"thead",{},[148,149,150,154,157,160,163],"tr",{},[151,152,153],"th",{},"Crawl",[151,155,156],{},"Concurrency",[151,158,159],{},"Throughput",[151,161,162],{},"Peak memory",[151,164,165],{},"Crawl file",[167,168,169,187,202,218,235,261],"tbody",{},[148,170,171,175,178,181,184],{},[172,173,174],"td",{},"2,000",[172,176,177],{},"1",[172,179,180],{},"3,900 URLs\u002Fsec",[172,182,183],{},"21 MiB",[172,185,186],{},"4.6 MiB",[148,188,189,191,194,197,200],{},[172,190,174],{},[172,192,193],{},"5",[172,195,196],{},"8,200 URLs\u002Fsec",[172,198,199],{},"22 MiB",[172,201,186],{},[148,203,204,206,209,212,215],{},[172,205,174],{},[172,207,208],{},"20",[172,210,211],{},"11,000 URLs\u002Fsec",[172,213,214],{},"25 MiB",[172,216,217],{},"4.7 MiB",[148,219,220,223,226,229,232],{},[172,221,222],{},"30,000",[172,224,225],{},"10",[172,227,228],{},"9,600 URLs\u002Fsec",[172,230,231],{},"111 MiB",[172,233,234],{},"69 MiB",[148,236,237,242,246,251,256],{},[172,238,239],{},[69,240,241],{},"50,000",[172,243,244],{},[69,245,208],{},[172,247,248],{},[69,249,250],{},"11,700 URLs\u002Fsec",[172,252,253],{},[69,254,255],{},"128 MiB",[172,257,258],{},[69,259,260],{},"100 MiB",[148,262,263,266,268,271,274],{},[172,264,265],{},"60,000",[172,267,225],{},[172,269,270],{},"10,600 URLs\u002Fsec",[172,272,273],{},"131 MiB",[172,275,276],{},"114 MiB",[11,278,279,280,282],{},"Our acceptance bar was 50,000 URLs in under roughly 500 MB. It uses ",[69,281,255],{},".",[11,284,285,286,289],{},"The row we care about most is the comparison between 30,000 and 60,000. Doubling the crawl costs ",[69,287,288],{},"1.19×"," the peak memory, or about 717 bytes of RAM per extra URL. On disk, a crawl costs about 2 KB per URL.",[11,291,292],{},"To be fair to the numbers: 717 bytes isn't zero, and we won't claim the line is perfectly flat. It's flat in the sense that matters: the cache fills up early and then stays put, and the growth after that is small enough that disk space and patience run out before memory does. We also only claim what we've measured.",[294,295,297],"callout",{"type":296},"note",[11,298,299],{},"Real sites won't crawl at 11,700 URLs a second, and that's by design. A real crawl is limited by the network and by politeness settings. With the default five concurrent requests and no crawl delay, expect tens of URLs per second against a typical server. Engine throughput matters because it means CrawlStat is never the slow part.",[29,301,303],{"id":302},"measuring-it-honestly","Measuring it honestly",[11,305,306],{},"Peak memory is easy to measure wrong. The usual measurement is a high water mark for the whole process, so two crawls run one after the other can't be compared: the second inherits the first one's peak. Each of our measurements runs in its own fresh process.",[11,308,309],{},"The flatness test does exactly that. It runs the crawler twice, at two crawl sizes, each in a fresh process, and compares the peaks. So flat memory is a design goal with a test behind it. If a change makes memory grow with crawl size, the test fails and the change doesn't ship.",[29,311,313],{"id":312},"the-tradeoff","The tradeoff",[11,315,316],{},"Nothing is free. Writing every URL to disk costs more than keeping it in a list in memory, and having a single writer puts an upper limit on how fast results can be saved. In practice, that limit is the throughput in the table above, around ten thousand URLs per second, far beyond what any polite crawl of a real server will reach. We pay some CPU so that crawl size never depends on RAM, and we think that's the right trade for a tool that runs on a laptop.",[11,318,319],{},"We get two bonuses out of it as well. Every crawl is a single file you can keep, copy or open later. And a crash only costs you the batch that hadn't been saved yet, not the whole afternoon.",[29,321,323],{"id":322},"try-it-on-something-big","Try it on something big",[11,325,326,327,332,333,282],{},"CrawlStat is free and open source, with no limit on how many URLs a crawl can fetch. Point it at the biggest site you're allowed to crawl and watch the memory graph stay flat. Start by ",[328,329,331],"a",{"href":330},"\u002Fdownload","downloading CrawlStat",", or read the crawl loop for yourself ",[328,334,338],{"href":335,"rel":336},"https:\u002F\u002Fgithub.com\u002Fclotairedamy\u002FCrawlStat",[337],"nofollow","on GitHub",{"title":340,"searchDepth":341,"depth":341,"links":342},"",2,[343,344,345,346,347,348,349],{"id":31,"depth":341,"text":32},{"id":60,"depth":341,"text":61},{"id":82,"depth":341,"text":83},{"id":136,"depth":341,"text":137},{"id":302,"depth":341,"text":303},{"id":312,"depth":341,"text":313},{"id":322,"depth":341,"text":323},null,"2026-10-14","CrawlStat has no crawl queue in memory. The crawl file is the queue, the list of seen URLs and the results, which is why 50,000 URLs fit in 128 MiB and doubling a crawl barely moves peak memory.","md",{},true,"\u002Fblog\u002Fflat-memory-crawling",6,{"title":5,"description":352},"blog\u002Fflat-memory-crawling","Engineering","SaZDNUwbcUubQMA2vXqsfIXqQ4z5PcYkjIv385lMj6E",[363,369,375,379,380,385],{"title":364,"path":365,"date":366,"tag":367,"readingTime":368},"CrawlStat is now free and open source","\u002Fblog\u002Fcrawlstat-is-open-source","2026-11-04","News",5,{"title":370,"path":371,"date":372,"tag":373,"readingTime":374},"Client reports, JavaScript rendering and a new way around a crawl","\u002Fblog\u002Fclient-reports-and-javascript-rendering","2026-10-26","Announcement",4,{"title":376,"path":377,"date":378,"tag":373,"readingTime":357},"Introducing CrawlStat","\u002Fblog\u002Fintroducing-crawlstat","2026-10-21",{"title":5,"path":356,"date":351,"tag":360,"readingTime":357},{"title":381,"path":382,"date":383,"tag":384,"readingTime":357},"Redirect chains: the chain is the finding","\u002Fblog\u002Fredirect-chains-the-chain-is-the-finding","2026-10-07","SEO guide",{"title":386,"path":387,"date":388,"tag":360,"readingTime":368},"Licensing without a database","\u002Fblog\u002Flicensing-without-a-database","2026-09-30",1793827598359]