[{"data":1,"prerenderedAt":560},["ShallowReactive",2],{"docs-\u002Fdocs\u002Fwatching-a-crawl":3,"docs-sib-\u002Fdocs\u002Fwatching-a-crawl":459,"docs-nav":534},{"id":4,"title":5,"body":6,"description":449,"extension":450,"meta":451,"navigation":452,"order":453,"path":454,"section":455,"seo":456,"stem":457,"__hash__":458},"docs\u002Fdocs\u002Fwatching-a-crawl.md","Watching a crawl",{"type":7,"value":8,"toc":436},"minimark",[9,14,33,39,45,48,138,142,145,185,199,203,206,226,239,243,249,252,256,267,271,340,350,353,380,384,397,410,414,426,430],[10,11,13],"h2",{"id":12},"the-live-view","The live view",[15,16,17,18,22,23,26,27,32],"p",{},"When you start a crawl, CrawlStat opens that crawl's page on its ",[19,20,21],"strong",{},"Overview"," tab. While the crawl runs, the Overview is the live view, titled ",[19,24,25],{},"Progress",": counters update as URLs are fetched, and a table shows the newest URLs as they come in. When the crawl is over, the Overview turns into the crawl's summary. See ",[28,29,31],"a",{"href":30},"\u002Fdocs\u002Fprojects#a-crawls-page","Projects"," for the rest of the page.",[34,35],"screenshot",{"alt":36,"caption":37,"src":38},"A crawl in progress: six counters, a speed gauge at 10.7 URLs a second, a progress bar and a live table of fetched URLs with status codes and response times","37 pages in, 742 discovered, 10.7 a second. Every URL lands in the live table the moment it answers.","\u002Fscreenshots\u002Fcrawl-progress.png",[15,40,41,42,44],{},"Under ",[19,43,25],{}," you'll see a status pill and the start URL. If the crawl was resumed, it says so. It also shows how many URLs were seeded at the start, which includes any URLs from a sitemap.",[15,46,47],{},"The status pill is one of:",[49,50,51,64],"table",{},[52,53,54],"thead",{},[55,56,57,61],"tr",{},[58,59,60],"th",{},"Status",[58,62,63],{},"Meaning",[65,66,67,78,88,98,108,118,128],"tbody",{},[55,68,69,75],{},[70,71,72],"td",{},[19,73,74],{},"Starting",[70,76,77],{},"Reading robots.txt and sitemaps and seeding the queue.",[55,79,80,85],{},[70,81,82],{},[19,83,84],{},"Crawling",[70,86,87],{},"Fetching URLs.",[55,89,90,95],{},[70,91,92],{},[19,93,94],{},"Paused",[70,96,97],{},"Waiting for you to resume. Nothing is being fetched.",[55,99,100,105],{},[70,101,102],{},[19,103,104],{},"Auditing",[70,106,107],{},"The crawl is over and the audits are running.",[55,109,110,115],{},[70,111,112],{},[19,113,114],{},"Finished",[70,116,117],{},"Everything is done.",[55,119,120,125],{},[70,121,122],{},[19,123,124],{},"Stopped",[70,126,127],{},"You stopped the crawl before it finished.",[55,129,130,135],{},[70,131,132],{},[19,133,134],{},"Failed",[70,136,137],{},"Something went wrong. The error is shown at the top of the screen.",[10,139,141],{"id":140},"the-counters","The counters",[15,143,144],{},"The tiles along the top count every URL the crawl knows about:",[146,147,148,155,161,167,173,179],"ul",{},[149,150,151,154],"li",{},[19,152,153],{},"Crawled:"," fetched and stored.",[149,156,157,160],{},[19,158,159],{},"Queued:"," waiting to be fetched.",[149,162,163,166],{},[19,164,165],{},"Discovered:"," every URL the crawl knows about so far, fetched or not.",[149,168,169,172],{},[19,170,171],{},"Client errors:"," URLs that returned a 4xx status, such as 404 Not Found.",[149,174,175,178],{},[19,176,177],{},"Server errors:"," URLs that returned a 5xx status, such as 500 or 503.",[149,180,181,184],{},[19,182,183],{},"Failed:"," URLs that couldn't be fetched at all, for example because of a timeout, a DNS failure or a refused connection.",[15,186,187,188,190,191,194,195,198],{},"When the audits have run, the ",[19,189,134],{}," tile becomes ",[19,192,193],{},"Issues",", showing the number of errors and warnings found. Click it to go straight to the ",[28,196,193],{"href":197},"\u002Fdocs\u002Fissues"," tab.",[10,200,202],{"id":201},"speed-and-progress","Speed and progress",[15,204,205],{},"The progress card shows:",[146,207,208,214,220],{},[149,209,210,213],{},[19,211,212],{},"URL\u002Fs:"," a gauge of how many URLs per second are being fetched, scaled against the fastest rate seen so far in this crawl.",[149,215,216,219],{},[19,217,218],{},"A progress bar."," Early on, CrawlStat can't know how big the site is, so the bar is indeterminate and says it's finding the size of the site. Once it has a sense of scale, it shows a percentage of what's known. That percentage can go down as new links are discovered. This is normal, because the site keeps getting bigger as CrawlStat finds more of it.",[149,221,222,225],{},[19,223,224],{},"Readouts"," for Discovered, Queued, In flight (requests happening right now), Elapsed time and Peak speed.",[15,227,228,229,233,234,238],{},"Speed depends mostly on the server and your connection. With the default of 5 simultaneous requests and no delay, expect tens of URLs per second from a typical site. A delay between requests, or a ",[230,231,232],"code",{},"Crawl-delay"," in the site's robots.txt, slows things down on purpose. See ",[28,235,237],{"href":236},"\u002Fdocs\u002Fsettings","Settings",".",[10,240,242],{"id":241},"warnings","Warnings",[15,244,245,246,248],{},"If anything unusual happens, a ",[19,247,242],{}," panel appears. It's collapsed by default, so click it to expand. Typical warnings include a sitemap that couldn't be fetched, a URL in a pasted list that couldn't be used, a redirect loop, or reaching your URL limit.",[15,250,251],{},"The panel shows the most recent 100 warnings. Every warning is also written to the crawl's own log inside the crawl file.",[10,253,255],{"id":254},"the-live-table","The live table",[15,257,258,261,262,266],{},[19,259,260],{},"Live URLs"," lists the most recently fetched URLs, newest first, with their status code, title, depth, response time and any error. To keep the screen fast, it holds only the latest 300. Nothing is lost: every URL is in the crawl file, and the ",[28,263,265],{"href":264},"\u002Fdocs\u002Fresults-table","Pages"," tab shows them all.",[10,268,270],{"id":269},"pause-resume-and-stop","Pause, resume and stop",[49,272,273,286],{},[52,274,275],{},[55,276,277,280,283],{},[58,278,279],{},"Action",[58,281,282],{},"How",[58,284,285],{},"What happens",[65,287,288,309,326],{},[55,289,290,295,306],{},[70,291,292],{},[19,293,294],{},"Pause",[70,296,297,298,300,301,305],{},"Click ",[19,299,294],{},", or press ",[302,303,304],"kbd",{},"⌘Space"," (macOS)",[70,307,308],{},"New requests stop. Requests already in flight finish.",[55,310,311,316,323],{},[70,312,313],{},[19,314,315],{},"Resume",[70,317,297,318,300,320,322],{},[19,319,315],{},[302,321,304],{}," again",[70,324,325],{},"The crawl carries on from where it paused.",[55,327,328,333,337],{},[70,329,330],{},[19,331,332],{},"Stop",[70,334,297,335],{},[19,336,332],{},[70,338,339],{},"Requests in flight finish, then the crawl ends.",[15,341,342,343,345,346,238],{},"A stopped crawl is still audited (if audits are on), so you get findings for everything fetched so far. The URLs that were still queued stay in the file, and you can carry on later with ",[19,344,315],{}," on the ",[28,347,349],{"href":348},"\u002Fdocs\u002Fprojects","Projects screen",[15,351,352],{},"When the crawl is over, the live view offers:",[146,354,355,364,374],{},[149,356,357,360,361,238],{},[19,358,359],{},"See pages",", which opens the complete ",[28,362,363],{"href":264},"Pages table",[149,365,366,369,370,238],{},[19,367,368],{},"Compare with previous crawl",", if you've crawled the same site before. It opens the comparison straight away. See ",[28,371,373],{"href":372},"\u002Fdocs\u002Fcomparing-crawls","Comparing crawls",[149,375,376,379],{},[19,377,378],{},"Clear",", which clears the live view. Clearing doesn't delete anything.",[10,381,383],{"id":382},"you-can-leave-the-screen","You can leave the screen",[15,385,386,387,389,390,393,394,396],{},"A crawl keeps running when you switch to another tab, another crawl or another screen. While it runs, the sidebar gains a tile for it, under ",[19,388,31],{},", with the site's name, a progress ring and a ",[19,391,392],{},"Live"," or ",[19,395,94],{}," badge. Click it to come back to the crawl from wherever you are.",[398,399,401],"callout",{"type":400},"tip",[15,402,403,404,406,407,409],{},"You can open the ",[19,405,265],{}," tab while a crawl is still running to look at what's been fetched so far. When the crawl finishes, click ",[19,408,359],{}," to load the complete set.",[10,411,413],{"id":412},"when-a-crawl-is-interrupted","When a crawl is interrupted",[15,415,416,417,419,420,422,423,425],{},"CrawlStat keeps its queue in the crawl file and saves results in small batches as it goes. If the app quits or your computer restarts in the middle of a crawl, you lose at most the last few URLs, and those are simply fetched again. Go to ",[19,418,31],{},", find the crawl, and click ",[19,421,315],{}," (or open the crawl and use ",[19,424,315],{}," in its header). CrawlStat picks up with the URLs that were still queued, using the same settings and limits the crawl started with.",[10,427,429],{"id":428},"reaching-the-url-limit","Reaching the URL limit",[15,431,432,433,435],{},"When a crawl reaches the URL limit you set, it stops fetching, finishes up and runs its audits as normal. A warning tells you how many discovered URLs were not crawled. Those URLs are recorded as skipped, so they won't be picked up by ",[19,434,315],{},". To crawl further, start a new crawl with a higher limit, or with no limit at all.",{"title":437,"searchDepth":438,"depth":438,"links":439},"",2,[440,441,442,443,444,445,446,447,448],{"id":12,"depth":438,"text":13},{"id":140,"depth":438,"text":141},{"id":201,"depth":438,"text":202},{"id":241,"depth":438,"text":242},{"id":254,"depth":438,"text":255},{"id":269,"depth":438,"text":270},{"id":382,"depth":438,"text":383},{"id":412,"depth":438,"text":413},{"id":428,"depth":438,"text":429},"What a crawl's Overview shows while it runs, what each counter means, and how to pause, resume, stop and pick up an interrupted crawl.","md",{},true,4,"\u002Fdocs\u002Fwatching-a-crawl","Using CrawlStat",{"title":5,"description":449},"docs\u002Fwatching-a-crawl","LIwY_DJHnnvgu8v9ilnrxV1PMrn3aj3Xy-daV9XdJhE",[460,464,467,471,474,478,479,482,484,488,492,496,500,502,504,506,510,514,518,522,526,530],{"title":461,"path":462,"order":463},"Client reports","\u002Fdocs\u002Fclient-reports",null,{"title":465,"path":466,"order":463},"UX testing","\u002Fdocs\u002Fux-testing",{"title":468,"path":469,"order":470},"Getting started","\u002Fdocs\u002Fgetting-started",1,{"title":472,"path":473,"order":438},"Installing CrawlStat","\u002Fdocs\u002Finstalling",{"title":475,"path":476,"order":477},"Your first crawl","\u002Fdocs\u002Fyour-first-crawl",3,{"title":5,"path":454,"order":453},{"title":480,"path":264,"order":481},"The Pages table",5,{"title":193,"path":197,"order":483},6,{"title":485,"path":486,"order":487},"The plan","\u002Fdocs\u002Fthe-plan",7,{"title":489,"path":490,"order":491},"Journeys","\u002Fdocs\u002Fjourneys",8,{"title":493,"path":494,"order":495},"Competitor Intelligence","\u002Fdocs\u002Fcompetitor-intelligence",8.7,{"title":497,"path":498,"order":499},"Accessibility","\u002Fdocs\u002Faccessibility",9,{"title":373,"path":372,"order":501},10,{"title":31,"path":348,"order":503},11,{"title":237,"path":236,"order":505},12,{"title":507,"path":508,"order":509},"Keyboard shortcuts","\u002Fdocs\u002Fkeyboard-shortcuts",13,{"title":511,"path":512,"order":513},"Open source and licence","\u002Fdocs\u002Flicensing",14,{"title":515,"path":516,"order":517},"Privacy and your data","\u002Fdocs\u002Fprivacy-and-data",15,{"title":519,"path":520,"order":521},"The crawler and robots.txt","\u002Fdocs\u002Fcrawler-and-robots",16,{"title":523,"path":524,"order":525},"Local AI","\u002Fdocs\u002Flocal-ai",17,{"title":527,"path":528,"order":529},"Troubleshooting","\u002Fdocs\u002Ftroubleshooting",18,{"title":531,"path":532,"order":533},"FAQ","\u002Fdocs\u002Ffaq",19,[535,536,537,539,540,541,542,543,544,545,546,547,548,549,550,551,552,554,555,556,557,559],{"title":461,"path":462,"section":463,"order":463},{"title":465,"path":466,"section":463,"order":463},{"title":468,"path":469,"section":538,"order":470},"Start here",{"title":472,"path":473,"section":538,"order":438},{"title":475,"path":476,"section":538,"order":477},{"title":5,"path":454,"section":455,"order":453},{"title":480,"path":264,"section":455,"order":481},{"title":193,"path":197,"section":455,"order":483},{"title":485,"path":486,"section":455,"order":487},{"title":489,"path":490,"section":455,"order":491},{"title":493,"path":494,"section":455,"order":495},{"title":497,"path":498,"section":455,"order":499},{"title":373,"path":372,"section":455,"order":501},{"title":31,"path":348,"section":455,"order":503},{"title":237,"path":236,"section":455,"order":505},{"title":507,"path":508,"section":455,"order":509},{"title":511,"path":512,"section":553,"order":513},"Account & data",{"title":515,"path":516,"section":553,"order":517},{"title":519,"path":520,"section":553,"order":521},{"title":523,"path":524,"section":553,"order":525},{"title":527,"path":528,"section":558,"order":529},"Help",{"title":531,"path":532,"section":558,"order":533},1793827598191]