--- title: "Technical SEO crawler | SEO Skill" description: "Check sitemap URL health first, then run a full crawl when you need page-level technical SEO evidence and every affected URL." canonical: "https://seoskill.dev/docs/crawler" language: "en" --- # Technical SEO crawler Check sitemap URL health first, then run a full crawl when you need page-level technical SEO evidence and every affected URL. The crawler has two stages. The fast health pass checks whether sitemap URLs respond cleanly. The full crawl fetches and analyses pages when you need a technical baseline, an affected URL set, or evidence to verify a change. ## Run the sitemap health pass first Give it the sitemap URL when you know it: ```sh seo crawl --sitemap-url https://example.com/sitemap.xml --health ``` This reads the sitemap, then checks each listed URL for status, redirects, robots decisions, network failures, and access blocks. It does not parse or render page bodies, join Google data, check external links, or write page and robots responses to the local cache. Requests begin one at a time and increase only after clean responses, up to the configured limit. When the sitemap URL is unknown, `--health` reads same-origin sitemap declarations from robots.txt and falls back to `/sitemap.xml`. An explicit `--sitemap-url` is faster and makes the checked scope unambiguous. ## Run the full crawl when you need page evidence Use the public URL you expect search engines and users to reach: ```sh seo crawl https://example.com --save ``` The default full crawl stops at 500 pages and a click depth of 16. It seeds URLs from sitemaps, follows internal links, respects robots.txt, and analyses page bodies. Saving the result lets focused reports reuse that evidence without fetching the site again. Use a project profile to join the crawl to available Search Console and Google Analytics evidence: ```sh seo crawl --project example --max-pages 500 --save ``` The live catalog exposes 51 documented crawler rules. List their stable IDs or inspect one rule’s evidence and repair guidance: ```sh seo rules seo explain --rule missing_meta_description ``` ## Keep the crawl focused and fast Bound the crawl to the question in front of you: ```sh seo crawl https://example.com --max-pages 100 --max-depth 4 seo crawl https://example.com --include '/products/' --exclude '/account/' seo crawl --sitemap-url https://example.com/sitemap.xml --health seo crawl --mode page --url https://example.com/pricing seo crawl --mode list --urls-file changed-urls.txt ``` `site` mode discovers internal URLs from the start page and sitemap. `sitemap` mode starts from a sitemap. `page` fetches one page, while `list` keeps the run to URLs you provide. When `robots.txt` declares same-origin sitemap files, the crawler tries those first and records which ones supplied URLs in JSON. If none return URLs, it also tries `/sitemap.xml`. `sitemapDiscovery.roots[].documents` keeps the response evidence for each fetched sitemap: HTTP status, content type, compression, byte counts, XML root, and any warning. Gzip-compressed sitemaps work. Malformed XML, a non-sitemap root, an unsuitable response, or a file over the 50 MiB protocol limit stays visible as partial or unavailable evidence rather than disappearing from the result. When a sitemap redirects, that document keeps both the requested URL and the final URL. The response status belongs to the final response. Treat a redirect as evidence to inspect, not proof that the sitemap is broken. `sitemapDiscovery.roots[].lastmods` records supplied `lastmod` values as unverified metadata. It counts values that parse and keeps small samples of malformed or future dates so you can inspect the generating system. The crawler never uses them to order discovery or claim that a page changed. An index-level `lastmod` describes the sitemap file, not every page it lists. Google only uses `lastmod` when it is consistently and verifiably accurate in its [sitemap guidance](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap). Google describes a sitemap as a discovery signal rather than an indexing guarantee in its [sitemap guidance](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview). Google limits each sitemap to 50 MB uncompressed or 50,000 URLs in its [sitemap build guide](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap). Treat sitemap membership the same way in a crawl report. A listed URL deserves inspection, but its presence does not prove that Google indexed it. ## External-link checks stay limited External-link checks run against a limited sample of the URLs returned by the crawl. The crawler spreads that sample across source pages, keeps each selected check on the page that linked to it, and records the limits, deferred URLs, and request failures in `externalLinkVerification` JSON. A 404 is evidence about the external target. It does not mean the page containing that link failed to crawl. ## Robots.txt controls crawling, not indexing The crawler skips disallowed URLs by default. That matches the site’s stated crawl policy and avoids surprising a site owner. ```sh seo crawl https://example.com --no-robots ``` Use `--no-robots` only when you control the target and need to diagnose rules that block your own crawl. Google notes that a disallowed URL can still appear in search results when it is discovered elsewhere. Its [robots.txt documentation](https://developers.google.com/search/docs/crawling-indexing/robots/intro) explains why robots.txt should not be treated as a removal tool. The crawler reports the rule it observed. It cannot know whether the rule is intentional. The same caution applies to `noindex`, canonicals, redirects, and snippet controls. ## Identify the crawler without opening a broad bypass Every HTTP and browser request uses a stable, versioned identity: ```text SEO-Skill/ (+https://seoskill.dev) ``` Robots rules are evaluated against the `SEO-Skill` token. JSON keeps the exact identity under `access.crawler`. If a request is challenged, denied, or rate limited, `access.samples` keeps the URL, status, provider indicators, request ID when available, and practical guidance. The crawler detects a Cloudflare Challenge Page from the response’s `cf-mitigated: challenge` header. Other Cloudflare headers show that the response passed through Cloudflare, but do not prove Cloudflare caused an ordinary 401, 403, or 429. Cloudflare documents the [Challenge response header](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/) and the available [custom Skip rules](https://developers.cloudflare.com/waf/custom-rules/skip/). A User-Agent can be spoofed, so never create a broad User-Agent-only allow rule. If the audit should have access, limit any temporary exception to the audit machine source IP, required hostname or paths, exact User-Agent, and only the security rule that blocked the request. Check Cloudflare Security Events first. If no event matches, inspect the origin security logs. ## Read technical findings without inventing an SEO rule The crawl records response status, redirects, robots directives, canonicals, titles, descriptions, headings, internal links, images, structured data, hreflang, viewport configuration, security headers, social previews, and available performance observations. Keep three distinctions in view: 1. **Observed evidence** says what the response or rendered page contained. 2. **A finding** applies a documented rule or heuristic to that evidence. 3. **An action** depends on the page’s purpose, intended canonical, template, and business context. A different canonical is not automatically wrong. Google treats a declared canonical as one signal and may select another URL. Its [canonical URL guidance](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) is the useful reference when a crawl finds conflicting signals. Structured data is similar. Valid markup can make a page eligible for a search appearance, but Google does not guarantee that appearance. Check the visible page, required properties, and the relevant [structured data policies](https://developers.google.com/search/docs/appearance/structured-data/sd-policies) before opening a ticket. ## Render JavaScript when the source response is not enough ```sh seo crawl https://example.com --rendering on --max-pages 50 ``` Use `--rendering on` when the links, canonicals, or main content are missing from the initial HTML. `auto` is the default. It only opens a browser when the response looks client-rendered. `off` keeps the raw HTTP response. The crawler uses a local Chrome or Chromium browser. It reuses a browser process during a crawl and keeps each page in its own browser context. If no compatible browser is available, the report says that rendered evidence is unavailable and keeps the raw response separate. It does not pretend raw HTML was rendered. Install Chrome or Chromium in the usual way. For a Playwright-managed browser, run the exact command shown by the CLI. `--js` remains an alias for `--rendering on` while existing scripts move to the explicit mode. When a browser renders a page, JSON output includes `fetchDiagnostics.rendering.documentDifference`. It shows whether the title, description, canonical, robots directives, headings, links, visible content, or structured data changed from the raw response. Saved crawls keep compact snapshots and text or link fingerprints rather than a second complete DOM. Google’s [JavaScript SEO guidance](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) explains how rendered HTML affects discovery and indexing. A local rendered crawl is still your observation, not a copy of Google’s renderer or index. ## Save a baseline and compare the next release ```sh seo crawl --project example --save seo crawl-reports --project example seo crawl-reports --project example --compare latest --against previous ``` The comparison reports pages and findings that were added, removed, fixed, or changed. Check the crawl limits and URL sets on both sides before calling a difference a regression. Comparing a 100-page sample with a 500-page crawl can produce dramatic numbers for a very boring reason. Run a fresh crawl after the deployment, then use the affected URLs to verify the template or route that changed. ## Export the part another person needs ```sh seo crawl https://example.com --format json --output crawl.json seo crawl https://example.com --format csv --csv issues --output issues.csv seo crawl https://example.com --format html --output report.html seo crawl https://example.com --format markdown --output tickets.md seo crawl --sitemap-url https://example.com/sitemap.xml --health --format junit --output sitemap-health.xml ``` JSON preserves the structured contract for scripts and agents. CSV works for sorting issues or pages. HTML gives you a local standalone report, while Markdown is useful for a limited implementation brief. JUnit creates one test case per sitemap document and listed URL with status, redirect, network, robots, and access-block evidence. ## Fail CI on a severity you have agreed to enforce ```sh seo crawl --sitemap-url https://example.com/sitemap.xml --health --format junit --output sitemap-health.xml --fail-on high ``` The process exits unsuccessfully when findings meet the selected severity or higher. JSON output still includes the triggering evidence. A CI gate is strongest when the crawl scope is stable and the team agrees what that severity means. Save the output as an artifact. Without the affected URLs and rule evidence, a red build is just a red build. Follow a failed health URL with a focused page audit or full crawl. Do not make every deployment download and analyse the whole site. Use the [CLI automation guide](https://seoskill.dev/docs/cli) for explicit JSON runs. The [AI search evidence guide](https://seoskill.dev/docs/ai-search) covers technical eligibility checks that reuse a saved crawl.