[{"data":1,"prerenderedAt":566},["ShallowReactive",2],{"docs-\u002Fdocs\u002Ftroubleshooting":3,"docs-sib-\u002Fdocs\u002Ftroubleshooting":461,"docs-nav":540},{"id":4,"title":5,"body":6,"description":451,"extension":452,"meta":453,"navigation":454,"order":455,"path":456,"section":457,"seo":458,"stem":459,"__hash__":460},"docs\u002Fdocs\u002Ftroubleshooting.md","Troubleshooting",{"type":7,"value":8,"toc":434},"minimark",[9,14,18,129,133,136,165,169,180,201,204,208,211,249,255,259,279,283,302,305,309,316,320,335,339,347,351,357,361,374,378,394,398,417],[10,11,13],"h2",{"id":12},"the-crawl-finds-only-one-page-or-very-few","The crawl finds only one page, or very few",[15,16,17],"p",{},"CrawlStat follows the links in the HTML the server sends. If it stops early, one of these is usually the cause:",[19,20,21,43,62,80,93,103,113,123],"ul",{},[22,23,24,28,29,32,33,36,37,42],"li",{},[25,26,27],"strong",{},"The site builds its links with JavaScript."," Unless you turn on rendering, CrawlStat reads the page as the server sent it. If the menu and links only appear after scripts run, there's nothing in that HTML to follow. Open the start URL's detail panel and check its ",[25,30,31],{},"Outlinks"," tab. If it's nearly empty, this is the likely reason: use ",[25,34,35],{},"Recrawl with changes…"," on the Projects screen and turn on ",[38,39,41],"a",{"href":40},"\u002Fdocs\u002Fyour-first-crawl#render-javascript","Render JavaScript",".",[22,44,45,48,49,53,54,57,58,42],{},[25,46,47],{},"robots.txt blocks CrawlStat."," Check the Internal tab for URLs marked as blocked, and the ",[38,50,52],{"href":51},"\u002Fdocs\u002Faudits\u002Fpages-blocked-by-robots-txt","Pages blocked by robots.txt"," issue. If it's your own site, you can turn off ",[25,55,56],{},"Respect robots.txt"," in ",[38,59,61],{"href":60},"\u002Fdocs\u002Fsettings","Settings",[22,63,64,67,68,72,73,76,77,42],{},[25,65,66],{},"The start URL redirects to another host."," If ",[69,70,71],"code",{},"example.com"," redirects to ",[69,74,75],{},"shop.example.com"," or a different domain, the destination counts as a different site. Start the crawl from the address the site actually ends up on, or turn on ",[25,78,79],{},"Crawl subdomains",[22,81,82,85,86,88,89,92],{},[25,83,84],{},"The links point at a subdomain."," With ",[25,87,79],{}," off, ",[69,90,91],{},"blog.example.com"," is treated as external and only checked, not crawled.",[22,94,95,98,99,102],{},[25,96,97],{},"Your patterns are too narrow."," If you used ",[25,100,101],{},"Only crawl URLs matching",", the URLs the start page links to must match one of your patterns.",[22,104,105,108,109,112],{},[25,106,107],{},"A depth limit is set."," A small ",[25,110,111],{},"Maximum depth"," stops the crawl close to the start URL.",[22,114,115,118,119,122],{},[25,116,117],{},"The links are nofollow."," Links marked ",[69,120,121],{},"rel=\"nofollow\""," aren't followed by default.",[22,124,125,128],{},[25,126,127],{},"The server is blocking the crawler."," See the next section.",[10,130,132],{"id":131},"lots-of-403s-429s-or-503s","Lots of 403s, 429s or 503s",[15,134,135],{},"These usually mean the server, or a firewall or bot protection service in front of it, is refusing CrawlStat or limiting how fast it can go:",[19,137,138,155],{},[22,139,140,143,144,146,147,150,151,154],{},[25,141,142],{},"429 Too Many Requests, or 503s that start part way through:"," you're crawling faster than the server allows. In ",[38,145,61],{"href":60},", lower ",[25,148,149],{},"Simultaneous requests"," to 1 or 2 and add a ",[25,152,153],{},"Delay between requests"," of 500 to 1000 ms, then crawl again.",[22,156,157,160,161,164],{},[25,158,159],{},"403 Forbidden on everything:"," a firewall may be blocking unfamiliar user agents. If it's your site, ask whoever runs it to allow ",[69,162,163],{},"CrawlStatBot",". Changing the user agent may also get through, but only do that on sites you control.",[10,166,168],{"id":167},"lots-of-failed-urls-or-timeouts","Lots of failed URLs or timeouts",[15,170,171,172,175,176,179],{},"URLs counted as ",[25,173,174],{},"Failed"," couldn't be fetched at all. The ",[25,177,178],{},"Error"," column on the Response codes tab says why.",[19,181,182,195],{},[22,183,184,187,188,191,192,194],{},[25,185,186],{},"Timeouts:"," the server is slow to respond. Raise ",[25,189,190],{},"Request timeout"," in Settings, and lower ",[25,193,149],{}," so the server isn't handling as many at once.",[22,196,197,200],{},[25,198,199],{},"Connection or DNS errors:"," check your internet connection, VPN or proxy. If only external links fail, the problem may be on those sites.",[15,202,203],{},"CrawlStat already retries timeouts and connection errors twice before marking a URL as failed.",[10,205,207],{"id":206},"the-crawl-is-slow","The crawl is slow",[15,209,210],{},"Crawl speed is mostly decided by the server. Things that slow it down:",[19,212,213,223,229,243],{},[22,214,215,222],{},[25,216,217,218,221],{},"A ",[69,219,220],{},"Crawl-delay"," in robots.txt."," CrawlStat honours it, and a delay of even a few seconds makes a big difference.",[22,224,225,228],{},[25,226,227],{},"Your own delay setting,"," or a low number of simultaneous requests.",[22,230,231,234,235,238,239,242],{},[25,232,233],{},"Slow pages."," Check the ",[25,236,237],{},"Time"," column for responses over a second, or the ",[25,240,241],{},"Slow"," lens in the Pages table.",[22,244,245,248],{},[25,246,247],{},"Render JavaScript is on."," Running every page in a browser makes a crawl several times slower. If the site's links and content are in the HTML the server sends, crawl without it.",[15,250,251,252,254],{},"On your own site, you can raise ",[25,253,149],{}," in Settings. Watch for server errors, because they mean it's too high.",[10,256,258],{"id":257},"the-crawl-stopped-before-the-end-of-the-site","The crawl stopped before the end of the site",[15,260,261,262,265,266,268,269,271,272,265,275,278],{},"CrawlStat has no URL limit of its own, so check the ",[25,263,264],{},"URL limit"," and ",[25,267,111],{}," you set in the New crawl dialog. A crawl that reaches your limit keeps everything it found up to that point, and a warning shows how many URLs weren't crawled. To crawl further, use ",[25,270,35],{}," and raise or clear the limit. If you want your limit to go further, turn off ",[25,273,274],{},"Images",[25,276,277],{},"Check external links",", or narrow the crawl with patterns.",[10,280,282],{"id":281},"the-issues-tab-is-empty","The Issues tab is empty",[15,284,285,286,289,290,293,294,297,298,301],{},"If a crawl's Issues tab says ",[25,287,288],{},"No issues found",", either the site is in good shape or the audits haven't run for this crawl. That happens when ",[25,291,292],{},"Run audits when a crawl finishes"," is off in Settings. Click ",[25,295,296],{},"Run the audits"," or ",[25,299,300],{},"Rerun audits"," to make sure.",[15,303,304],{},"Some issues are deliberately not reported for certain pages. For example, title and description checks skip pages that can't be indexed, such as noindex pages, redirects and error pages.",[10,306,308],{"id":307},"render-javascript-is-greyed-out","Render JavaScript is greyed out",[15,310,311,312,315],{},"CrawlStat renders pages with a browser you already have: Chrome, Chromium, Edge or Brave. If it can't find one, the switch is greyed out and says so. Install one of them and reopen the New crawl dialog. If yours is installed somewhere unusual, set the ",[69,313,314],{},"CRAWLSTAT_CHROME"," environment variable to the browser's full path before starting CrawlStat.",[10,317,319],{"id":318},"the-pdf-option-is-greyed-out-when-exporting-a-report","The PDF option is greyed out when exporting a report",[15,321,322,326,327,330,331,334],{},[38,323,325],{"href":324},"\u002Fdocs\u002Fclient-reports","Client reports"," are printed to PDF by Chrome, Edge or Brave, and the PDF option needs one of them installed. Without one, choose ",[25,328,329],{},"Web page",": it's the same report as a single file, and any browser's ",[25,332,333],{},"Print"," dialog turns it into an A4 PDF.",[10,336,338],{"id":337},"the-client-report-has-no-progress-section","The client report has no progress section",[15,340,341,342,346],{},"The progress section compares the crawl with an earlier crawl of the same site, so it's only offered when there is one. Recrawl the site from ",[38,343,345],{"href":344},"\u002Fdocs\u002Fprojects#recrawling","Projects",", then export the report from the new crawl.",[10,348,350],{"id":349},"crawlstat-is-not-allowed-to-write-to-when-exporting","\"CrawlStat is not allowed to write to…\" when exporting",[15,352,353,354,42],{},"Exports are written to your Downloads folder (or your home folder if there's no Downloads folder). This message means the operating system refused to let CrawlStat write there. Check that the folder exists and that you have permission to write to it, then try again. On macOS, check whether CrawlStat has been denied access to Downloads in ",[25,355,356],{},"System Settings → Privacy & Security → Files and Folders",[10,358,360],{"id":359},"a-crawl-has-disappeared-from-projects","A crawl has disappeared from Projects",[15,362,363,364,367,368,370,371,42],{},"The Projects screen lists your 20 most recent crawls. Crawls also drop off the list if their file has been moved or deleted. If you still have the file, put it back in CrawlStat's ",[69,365,366],{},"projects"," folder. You can find the path in the Projects header and in ",[38,369,61],{"href":60},". Then click ",[25,372,373],{},"Refresh",[10,375,377],{"id":376},"a-crawl-was-interrupted","A crawl was interrupted",[15,379,380,381,383,384,387,388,390,391,42],{},"If CrawlStat quit or your computer restarted during a crawl, go to ",[25,382,345],{}," and click ",[25,385,386],{},"Resume"," on that crawl's card, or open the crawl and click ",[25,389,386],{}," in its header. It picks up where it left off. See ",[38,392,345],{"href":393},"\u002Fdocs\u002Fprojects#resuming-a-crawl",[10,395,397],{"id":396},"still-stuck","Still stuck?",[15,399,400,406,407,411,412,416],{},[38,401,405],{"href":402,"rel":403},"https:\u002F\u002Fgithub.com\u002Fclotairedamy\u002FCrawlStat\u002Fissues",[404],"nofollow","Open an issue on GitHub",", or ",[38,408,410],{"href":409},"\u002Fcontact","send us a message"," and choose ",[413,414,415],"em",{},"Something is not working",". It helps to include:",[19,418,419,422,428,431],{},[22,420,421],{},"Your operating system and its version.",[22,423,424,425,42],{},"The engine version from ",[25,426,427],{},"Settings → About",[22,429,430],{},"What you were doing, and the exact text of any error or warning.",[22,432,433],{},"If it's about a particular site, its address, if you're able to share it.",{"title":435,"searchDepth":436,"depth":436,"links":437},"",2,[438,439,440,441,442,443,444,445,446,447,448,449,450],{"id":12,"depth":436,"text":13},{"id":131,"depth":436,"text":132},{"id":167,"depth":436,"text":168},{"id":206,"depth":436,"text":207},{"id":257,"depth":436,"text":258},{"id":281,"depth":436,"text":282},{"id":307,"depth":436,"text":308},{"id":318,"depth":436,"text":319},{"id":337,"depth":436,"text":338},{"id":349,"depth":436,"text":350},{"id":359,"depth":436,"text":360},{"id":376,"depth":436,"text":377},{"id":396,"depth":436,"text":397},"Fixes for the most common problems, from crawls that find too few pages or hit lots of errors to empty issue lists, report exports and failed exports.","md",{},true,18,"\u002Fdocs\u002Ftroubleshooting","Help",{"title":5,"description":451},"docs\u002Ftroubleshooting","1G2vz1TO2FLL8Dp25cUUbTyaCmLlj88vcNHI7xmlLas",[462,464,467,471,474,478,482,486,490,494,498,502,506,510,513,515,519,523,527,531,535,536],{"title":325,"path":324,"order":463},null,{"title":465,"path":466,"order":463},"UX testing","\u002Fdocs\u002Fux-testing",{"title":468,"path":469,"order":470},"Getting started","\u002Fdocs\u002Fgetting-started",1,{"title":472,"path":473,"order":436},"Installing CrawlStat","\u002Fdocs\u002Finstalling",{"title":475,"path":476,"order":477},"Your first crawl","\u002Fdocs\u002Fyour-first-crawl",3,{"title":479,"path":480,"order":481},"Watching a crawl","\u002Fdocs\u002Fwatching-a-crawl",4,{"title":483,"path":484,"order":485},"The Pages table","\u002Fdocs\u002Fresults-table",5,{"title":487,"path":488,"order":489},"Issues","\u002Fdocs\u002Fissues",6,{"title":491,"path":492,"order":493},"The plan","\u002Fdocs\u002Fthe-plan",7,{"title":495,"path":496,"order":497},"Journeys","\u002Fdocs\u002Fjourneys",8,{"title":499,"path":500,"order":501},"Competitor Intelligence","\u002Fdocs\u002Fcompetitor-intelligence",8.7,{"title":503,"path":504,"order":505},"Accessibility","\u002Fdocs\u002Faccessibility",9,{"title":507,"path":508,"order":509},"Comparing crawls","\u002Fdocs\u002Fcomparing-crawls",10,{"title":345,"path":511,"order":512},"\u002Fdocs\u002Fprojects",11,{"title":61,"path":60,"order":514},12,{"title":516,"path":517,"order":518},"Keyboard shortcuts","\u002Fdocs\u002Fkeyboard-shortcuts",13,{"title":520,"path":521,"order":522},"Open source and licence","\u002Fdocs\u002Flicensing",14,{"title":524,"path":525,"order":526},"Privacy and your data","\u002Fdocs\u002Fprivacy-and-data",15,{"title":528,"path":529,"order":530},"The crawler and robots.txt","\u002Fdocs\u002Fcrawler-and-robots",16,{"title":532,"path":533,"order":534},"Local AI","\u002Fdocs\u002Flocal-ai",17,{"title":5,"path":456,"order":455},{"title":537,"path":538,"order":539},"FAQ","\u002Fdocs\u002Ffaq",19,[541,542,543,545,546,547,549,550,551,552,553,554,555,556,557,558,559,561,562,563,564,565],{"title":325,"path":324,"section":463,"order":463},{"title":465,"path":466,"section":463,"order":463},{"title":468,"path":469,"section":544,"order":470},"Start here",{"title":472,"path":473,"section":544,"order":436},{"title":475,"path":476,"section":544,"order":477},{"title":479,"path":480,"section":548,"order":481},"Using CrawlStat",{"title":483,"path":484,"section":548,"order":485},{"title":487,"path":488,"section":548,"order":489},{"title":491,"path":492,"section":548,"order":493},{"title":495,"path":496,"section":548,"order":497},{"title":499,"path":500,"section":548,"order":501},{"title":503,"path":504,"section":548,"order":505},{"title":507,"path":508,"section":548,"order":509},{"title":345,"path":511,"section":548,"order":512},{"title":61,"path":60,"section":548,"order":514},{"title":516,"path":517,"section":548,"order":518},{"title":520,"path":521,"section":560,"order":522},"Account & data",{"title":524,"path":525,"section":560,"order":526},{"title":528,"path":529,"section":560,"order":530},{"title":532,"path":533,"section":560,"order":534},{"title":5,"path":456,"section":457,"order":455},{"title":537,"path":538,"section":457,"order":539},1793827598175]