[{"data":1,"prerenderedAt":595},["ShallowReactive",2],{"docs-\u002Fdocs\u002Fyour-first-crawl":3,"docs-sib-\u002Fdocs\u002Fyour-first-crawl":501,"docs-nav":569},{"id":4,"title":5,"body":6,"description":492,"extension":493,"meta":494,"navigation":495,"order":483,"path":496,"section":497,"seo":498,"stem":499,"__hash__":500},"docs\u002Fdocs\u002Fyour-first-crawl.md","Your first crawl",{"type":7,"value":8,"toc":476},"minimark",[9,14,20,24,61,72,92,96,158,162,194,197,205,208,222,225,232,235,239,242,256,259,262,268,294,302,312,316,321,335,338,348,351,357,360,367,377,381,398,405,409,424,428],[10,11,13],"h2",{"id":12},"open-the-new-crawl-dialog","Open the New crawl dialog",[15,16],"screenshot",{"alt":17,"caption":18,"src":19},"The New crawl dialog in dark mode: Spider, Sitemap and List modes, a start URL, a URL limit, maximum depth and a Render JavaScript switch","Pick a mode, paste a URL, press Start. Everything else has a sensible default.","\u002Fscreenshots\u002Fnew-crawl-dark-ember.png",[21,22,23],"p",{},"There are three ways to open it:",[25,26,27,40,54],"ul",{},[28,29,30,31,35,36,39],"li",{},"Click ",[32,33,34],"strong",{},"New crawl",": the ",[32,37,38],{},"+"," at the top of the sidebar, or the button in the Projects header.",[28,41,42,43,47,48,53],{},"Press ",[44,45,46],"kbd",{},"⌘N"," on macOS. See ",[49,50,52],"a",{"href":51},"\u002Fdocs\u002Fkeyboard-shortcuts","Keyboard shortcuts"," for Windows and Linux.",[28,55,56,57,60],{},"On the Projects screen, when you have no crawls yet, click ",[32,58,59],{},"Start your first crawl",".",[21,62,63,64,67,68,71],{},"The dialog has a mode selector at the top, an address field, two limits, a ",[32,65,66],{},"Render JavaScript"," switch and, under ",[32,69,70],{},"More options",", URL patterns.",[73,74,76],"callout",{"type":75},"tip",[21,77,78,79,82,83,87,88,91],{},"To crawl a site again with the same settings, you don't need this dialog: use ",[32,80,81],{},"Recrawl"," on its card in ",[49,84,86],{"href":85},"\u002Fdocs\u002Fprojects","Projects",". ",[32,89,90],{},"Recrawl with changes…"," opens this dialog filled in with the earlier crawl's settings.",[10,93,95],{"id":94},"choose-a-mode","Choose a mode",[97,98,99,115],"table",{},[100,101,102],"thead",{},[103,104,105,109,112],"tr",{},[106,107,108],"th",{},"Mode",[106,110,111],{},"What it does",[106,113,114],{},"Use it when",[116,117,118,132,145],"tbody",{},[103,119,120,126,129],{},[121,122,123],"td",{},[32,124,125],{},"Spider",[121,127,128],{},"Starts at one URL and follows links outwards across the site.",[121,130,131],{},"You want to see a site the way a search engine discovers it. This is the usual choice.",[103,133,134,139,142],{},[121,135,136],{},[32,137,138],{},"Sitemap",[121,140,141],{},"Starts from every URL a sitemap declares, then follows links from those pages.",[121,143,144],{},"You want to make sure everything you've told search engines about is crawled, for example to find sitemap URLs that redirect or return errors.",[103,146,147,152,155],{},[121,148,149],{},[32,150,151],{},"List",[121,153,154],{},"Fetches exactly the URLs you paste, and nothing else.",[121,156,157],{},"You have a specific set of pages to check, such as URLs from a migration spreadsheet.",[159,160,125],"h3",{"id":161},"spider",[21,163,164,165,168,169,173,174,177,178,181,182,185,186,189,190,60],{},"Type a ",[32,166,167],{},"Start URL",". You can leave out the scheme: ",[170,171,172],"code",{},"example.com"," works, as does ",[170,175,176],{},"https:\u002F\u002Fwww.example.com\u002Fblog\u002F",". CrawlStat crawls outwards from there, staying on the same site. ",[170,179,180],{},"www."," and the bare domain count as the same site. Subdomains such as ",[170,183,184],{},"blog.example.com"," are left out unless you turn on ",[32,187,188],{},"Crawl subdomains"," in ",[49,191,193],{"href":192},"\u002Fdocs\u002Fsettings","Settings",[21,195,196],{},"Links to other sites are checked once to see whether they work, but never followed.",[21,198,199,200,204],{},"If the site's robots.txt advertises a sitemap, CrawlStat reads that too and adds its URLs to the crawl. That's how it finds pages that are in your sitemap but that nothing links to (see ",[49,201,203],{"href":202},"\u002Fdocs\u002Faudits\u002Forphan-pages","Orphan pages",").",[159,206,138],{"id":207},"sitemap",[21,209,164,210,213,214,217,218,221],{},[32,211,212],{},"Sitemap URL",", for example ",[170,215,216],{},"https:\u002F\u002Fexample.com\u002Fsitemap.xml",". Sitemap indexes, which list other sitemaps, are followed. Gzipped sitemaps (",[170,219,220],{},".xml.gz",") are handled too. Every URL in the sitemap is queued, and links found on those pages are followed as in a Spider crawl.",[159,223,151],{"id":224},"list",[21,226,227,228,231],{},"Paste your URLs into ",[32,229,230],{},"URLs to fetch",", one per line. Blank lines are ignored. In this mode links aren't followed: CrawlStat records what each page links to but doesn't crawl those links. Redirects are still followed, so you can see where each URL ends up.",[21,233,234],{},"If some lines aren't usable URLs, CrawlStat skips them and lists them as warnings on the Crawl screen. If none of them are usable, the crawl won't start and the dialog tells you why.",[10,236,238],{"id":237},"set-limits","Set limits",[21,240,241],{},"Both limits are optional:",[25,243,244,250],{},[28,245,246,249],{},[32,247,248],{},"URL limit:"," the most URLs the crawl will fetch. Leave it blank for no limit.",[28,251,252,255],{},[32,253,254],{},"Maximum depth:"," how many clicks from the start URL the crawl will go. The start URL is depth 0, pages it links to are depth 1, and so on. Leave it blank for no limit.",[21,257,258],{},"A depth limit is a quick way to sample a large site: a depth of 3 shows you the pages that matter most to its navigation.",[10,260,66],{"id":261},"render-javascript",[21,263,264,265,267],{},"Some sites send an almost empty page and build everything in the browser with JavaScript: React, Vue, Angular and similar. Crawled as served, they look like one blank page with no links. Turn on ",[32,266,66],{}," and CrawlStat runs each page in a real browser first, then reads the page as a visitor would see it: the title, headings, text and links that the scripts put there.",[25,269,270,276,282,288],{},[28,271,272,275],{},[32,273,274],{},"It uses a browser you already have."," Chrome, Chromium, Edge or Brave. Nothing is downloaded. The switch names the browser it found. If there isn't one, the switch is greyed out and says so.",[28,277,278,281],{},[32,279,280],{},"The server's answer still counts."," The status code, headers, redirects and response time come from the normal request, exactly as without rendering. Only the page's content comes from the browser.",[28,283,284,287],{},[32,285,286],{},"It's several times slower."," Each page opens in its own browser, which closes straight after, four at a time, with up to four seconds for its scripts to run. Images aren't loaded during rendering.",[28,289,290,293],{},[32,291,292],{},"It never loses a page."," If the browser fails on a page, CrawlStat keeps the HTML the server sent instead, so the crawl carries on.",[21,295,296,297,301],{},"It's off by default, because most sites don't need it and crawls are much faster without it. If you're not sure, crawl once without it: if pages show almost no words and almost no links, turn it on and recrawl. A ",[49,298,300],{"href":299},"\u002Fdocs\u002Fclient-reports","client report"," notes on its cover when JavaScript was rendered.",[73,303,305],{"type":304},"note",[21,306,307,308,311],{},"If your browser is installed somewhere unusual, set the ",[170,309,310],{},"CRAWLSTAT_CHROME"," environment variable to its full path before starting CrawlStat.",[10,313,315],{"id":314},"narrow-the-crawl-with-patterns","Narrow the crawl with patterns",[21,317,30,318,320],{},[32,319,70],{}," to restrict which URLs are crawled. Both fields take regular expressions, one per line:",[25,322,323,329],{},[28,324,325,328],{},[32,326,327],{},"Only crawl URLs matching:"," if you add any patterns, a URL must match at least one of them to be crawled. Leave it blank to crawl everything.",[28,330,331,334],{},[32,332,333],{},"Never crawl URLs matching:"," any URL matching one of these is skipped. These rules are applied after the include rules, so an exclude always wins.",[21,336,337],{},"Patterns are matched against the full URL, so it helps to anchor them. The URL you start from is always fetched. The patterns apply to everything found after it, including sitemap entries and links to other sites. Some examples:",[339,340,346],"pre",{"className":341,"code":343,"language":344,"meta":345},[342],"language-text","^https:\u002F\u002Fexample\\.com\u002Fblog\u002F\n","text","",[170,347,343],{"__ignoreMap":345},[21,349,350],{},"Only crawls the blog. Because links to other sites don't match this pattern either, they won't be checked.",[339,352,355],{"className":353,"code":354,"language":344,"meta":345},[342],"\u002Fcart\n\\?page=\n\u002Ftag\u002F\n",[170,356,354],{"__ignoreMap":345},[21,358,359],{},"Skips the cart, paginated listings and tag archives. Each line is a separate pattern.",[21,361,362,363,366],{},"If a pattern isn't a valid regular expression, the dialog shows the error when you click ",[32,364,365],{},"Start crawl",", and nothing is started.",[21,368,369,370,373,374,376],{},"Below the patterns, a summary line shows the defaults this crawl will use. For example: \"5 at a time · robots.txt respected · external links checked · audits on finish\". Click ",[32,371,372],{},"Change"," to go to ",[49,375,193],{"href":192}," and adjust them. Settings apply to new crawls only.",[10,378,380],{"id":379},"start-the-crawl","Start the crawl",[21,382,30,383,385,386,389,390,393,394,60],{},[32,384,365],{},", or press ",[44,387,388],{},"Enter"," in the address field. CrawlStat creates a new crawl file and takes you to that crawl's page, where the ",[32,391,392],{},"Overview"," tab shows it running live. See ",[49,395,397],{"href":396},"\u002Fdocs\u002Fwatching-a-crawl","Watching a crawl",[21,399,400,401,404],{},"The bottom of the dialog shows where the file is saved. Each crawl gets its own file, named after the site and the time it started, for example ",[170,402,403],{},"example-com-20260224-1432.db",". Crawling the same site again creates a new file rather than overwriting the old one, so you keep a history.",[10,406,408],{"id":407},"before-you-crawl-someone-elses-site","Before you crawl someone else's site",[21,410,411,412,415,416,419,420,60],{},"CrawlStat is polite by default. It reads robots.txt and obeys it, honours any ",[170,413,414],{},"Crawl-delay"," it asks for, and identifies itself as ",[170,417,418],{},"CrawlStatBot",". If you're crawling a site you don't run, keep those defaults, and consider adding a delay between requests if the site seems slow. See ",[49,421,423],{"href":422},"\u002Fdocs\u002Fcrawler-and-robots","The crawler and robots.txt",[10,425,427],{"id":426},"what-happens-next","What happens next",[429,430,431,436,439,470],"ol",{},[28,432,433,434,60],{},"The crawl runs. You can pause, resume or stop it at any time. See ",[49,435,397],{"href":396},[28,437,438],{},"When it finishes, CrawlStat runs its audits automatically.",[28,440,441,442,444,445,448,449,452,453,456,457,461,462,465,466,204],{},"The crawl's page fills in: ",[32,443,392],{}," for the headline figures, ",[32,446,447],{},"Issues"," for what needs fixing (",[49,450,447],{"href":451},"\u002Fdocs\u002Fissues","), ",[32,454,455],{},"Pages"," to explore every URL (",[49,458,460],{"href":459},"\u002Fdocs\u002Fresults-table","the Pages table","), and ",[32,463,464],{},"Plan"," for what to fix first (",[49,467,469],{"href":468},"\u002Fdocs\u002Fthe-plan","The plan",[28,471,472,473,60],{},"When you're ready to share it, ",[49,474,475],{"href":299},"export a client report",{"title":345,"searchDepth":477,"depth":477,"links":478},2,[479,480,486,487,488,489,490,491],{"id":12,"depth":477,"text":13},{"id":94,"depth":477,"text":95,"children":481},[482,484,485],{"id":161,"depth":483,"text":125},3,{"id":207,"depth":483,"text":138},{"id":224,"depth":483,"text":151},{"id":237,"depth":477,"text":238},{"id":261,"depth":477,"text":66},{"id":314,"depth":477,"text":315},{"id":379,"depth":477,"text":380},{"id":407,"depth":477,"text":408},{"id":426,"depth":477,"text":427},"Start a crawl from the New crawl dialog, choose between Spider, Sitemap and List modes, render JavaScript for sites built with scripts, and use limits and URL patterns to crawl exactly what you need.","md",{},true,"\u002Fdocs\u002Fyour-first-crawl","Start here",{"title":5,"description":492},"docs\u002Fyour-first-crawl","fqWtB_DJy9zTyC4T7q19vuXQYQyHM7nlS19-MKqvTuA",[502,505,508,512,515,516,518,521,523,525,529,533,537,541,543,545,547,551,555,557,561,565],{"title":503,"path":299,"order":504},"Client reports",null,{"title":506,"path":507,"order":504},"UX testing","\u002Fdocs\u002Fux-testing",{"title":509,"path":510,"order":511},"Getting started","\u002Fdocs\u002Fgetting-started",1,{"title":513,"path":514,"order":477},"Installing CrawlStat","\u002Fdocs\u002Finstalling",{"title":5,"path":496,"order":483},{"title":397,"path":396,"order":517},4,{"title":519,"path":459,"order":520},"The Pages table",5,{"title":447,"path":451,"order":522},6,{"title":469,"path":468,"order":524},7,{"title":526,"path":527,"order":528},"Journeys","\u002Fdocs\u002Fjourneys",8,{"title":530,"path":531,"order":532},"Competitor Intelligence","\u002Fdocs\u002Fcompetitor-intelligence",8.7,{"title":534,"path":535,"order":536},"Accessibility","\u002Fdocs\u002Faccessibility",9,{"title":538,"path":539,"order":540},"Comparing crawls","\u002Fdocs\u002Fcomparing-crawls",10,{"title":86,"path":85,"order":542},11,{"title":193,"path":192,"order":544},12,{"title":52,"path":51,"order":546},13,{"title":548,"path":549,"order":550},"Open source and licence","\u002Fdocs\u002Flicensing",14,{"title":552,"path":553,"order":554},"Privacy and your data","\u002Fdocs\u002Fprivacy-and-data",15,{"title":423,"path":422,"order":556},16,{"title":558,"path":559,"order":560},"Local AI","\u002Fdocs\u002Flocal-ai",17,{"title":562,"path":563,"order":564},"Troubleshooting","\u002Fdocs\u002Ftroubleshooting",18,{"title":566,"path":567,"order":568},"FAQ","\u002Fdocs\u002Ffaq",19,[570,571,572,573,574,575,577,578,579,580,581,582,583,584,585,586,587,589,590,591,592,594],{"title":503,"path":299,"section":504,"order":504},{"title":506,"path":507,"section":504,"order":504},{"title":509,"path":510,"section":497,"order":511},{"title":513,"path":514,"section":497,"order":477},{"title":5,"path":496,"section":497,"order":483},{"title":397,"path":396,"section":576,"order":517},"Using CrawlStat",{"title":519,"path":459,"section":576,"order":520},{"title":447,"path":451,"section":576,"order":522},{"title":469,"path":468,"section":576,"order":524},{"title":526,"path":527,"section":576,"order":528},{"title":530,"path":531,"section":576,"order":532},{"title":534,"path":535,"section":576,"order":536},{"title":538,"path":539,"section":576,"order":540},{"title":86,"path":85,"section":576,"order":542},{"title":193,"path":192,"section":576,"order":544},{"title":52,"path":51,"section":576,"order":546},{"title":548,"path":549,"section":588,"order":550},"Account & data",{"title":552,"path":553,"section":588,"order":554},{"title":423,"path":422,"section":588,"order":556},{"title":558,"path":559,"section":588,"order":560},{"title":562,"path":563,"section":593,"order":564},"Help",{"title":566,"path":567,"section":593,"order":568},1793827598147]