Technical SEO crawler

Check sitemap URL health first, then run a full crawl when you need page-level technical SEO evidence and every affected URL.

The crawler has two stages. The fast health pass checks whether sitemap URLs respond cleanly. The full crawl fetches and analyses pages when you need a technical baseline, an affected URL set, or evidence to verify a change.

Run the sitemap health pass first

Give it the sitemap URL when you know it:

seo crawl --sitemap-url https://example.com/sitemap.xml --health

This reads the sitemap, then checks each listed URL for status, redirects, robots decisions, network failures, and access blocks. It does not parse or render page bodies, join Google data, check external links, or write page and robots responses to the local cache. Requests begin one at a time and increase only after clean responses, up to the configured limit.

When the sitemap URL is unknown, --health reads same-origin sitemap declarations from robots.txt and falls back to /sitemap.xml. An explicit --sitemap-url is faster and makes the checked scope unambiguous.

Run the full crawl when you need page evidence

Use the public URL you expect search engines and users to reach:

seo crawl https://example.com --save

The default full crawl stops at 500 pages and a click depth of 16. It seeds URLs from sitemaps, follows internal links, respects robots.txt, and analyses page bodies. Saving the result lets focused reports reuse that evidence without fetching the site again.

Use a project profile to join the crawl to available Search Console and Google Analytics evidence:

seo crawl --project example --max-pages 500 --save

The live catalog exposes 51 documented crawler rules. List their stable IDs or inspect one rule’s evidence and repair guidance:

seo rules
seo explain --rule missing_meta_description

Keep the crawl focused and fast

Bound the crawl to the question in front of you:

seo crawl https://example.com --max-pages 100 --max-depth 4
seo crawl https://example.com --include '/products/' --exclude '/account/'
seo crawl --sitemap-url https://example.com/sitemap.xml --health
seo crawl --mode page --url https://example.com/pricing
seo crawl --mode list --urls-file changed-urls.txt

site mode discovers internal URLs from the start page and sitemap. sitemap mode starts from a sitemap. page fetches one page, while list keeps the run to URLs you provide. When robots.txt declares same-origin sitemap files, the crawler tries those first and records which ones supplied URLs in JSON. If none return URLs, it also tries /sitemap.xml.

sitemapDiscovery.roots[].documents keeps the response evidence for each fetched sitemap: HTTP status, content type, compression, byte counts, XML root, and any warning. Gzip-compressed sitemaps work. Malformed XML, a non-sitemap root, an unsuitable response, or a file over the 50 MiB protocol limit stays visible as partial or unavailable evidence rather than disappearing from the result.

When a sitemap redirects, that document keeps both the requested URL and the final URL. The response status belongs to the final response. Treat a redirect as evidence to inspect, not proof that the sitemap is broken.

sitemapDiscovery.roots[].lastmods records supplied lastmod values as unverified metadata. It counts values that parse and keeps small samples of malformed or future dates so you can inspect the generating system. The crawler never uses them to order discovery or claim that a page changed. An index-level lastmod describes the sitemap file, not every page it lists. Google only uses lastmod when it is consistently and verifiably accurate in its sitemap guidance.

Google describes a sitemap as a discovery signal rather than an indexing guarantee in its sitemap guidance. Google limits each sitemap to 50 MB uncompressed or 50,000 URLs in its sitemap build guide. Treat sitemap membership the same way in a crawl report. A listed URL deserves inspection, but its presence does not prove that Google indexed it.

External-link checks stay limited

External-link checks run against a limited sample of the URLs returned by the crawl. The crawler spreads that sample across source pages, keeps each selected check on the page that linked to it, and records the limits, deferred URLs, and request failures in externalLinkVerification JSON. A 404 is evidence about the external target. It does not mean the page containing that link failed to crawl.

Robots.txt controls crawling, not indexing

The crawler skips disallowed URLs by default. That matches the site’s stated crawl policy and avoids surprising a site owner.

seo crawl https://example.com --no-robots

Use --no-robots only when you control the target and need to diagnose rules that block your own crawl. Google notes that a disallowed URL can still appear in search results when it is discovered elsewhere. Its robots.txt documentation explains why robots.txt should not be treated as a removal tool.

The crawler reports the rule it observed. It cannot know whether the rule is intentional. The same caution applies to noindex, canonicals, redirects, and snippet controls.

Identify the crawler without opening a broad bypass

Every HTTP and browser request uses a stable, versioned identity:

SEO-Skill/<version> (+https://seoskill.dev)

Robots rules are evaluated against the SEO-Skill token. JSON keeps the exact identity under access.crawler. If a request is challenged, denied, or rate limited, access.samples keeps the URL, status, provider indicators, request ID when available, and practical guidance.

The crawler detects a Cloudflare Challenge Page from the response’s cf-mitigated: challenge header. Other Cloudflare headers show that the response passed through Cloudflare, but do not prove Cloudflare caused an ordinary 401, 403, or 429. Cloudflare documents the Challenge response header and the available custom Skip rules.

A User-Agent can be spoofed, so never create a broad User-Agent-only allow rule. If the audit should have access, limit any temporary exception to the audit machine source IP, required hostname or paths, exact User-Agent, and only the security rule that blocked the request. Check Cloudflare Security Events first. If no event matches, inspect the origin security logs.

Read technical findings without inventing an SEO rule

The crawl records response status, redirects, robots directives, canonicals, titles, descriptions, headings, internal links, images, structured data, hreflang, viewport configuration, security headers, social previews, and available performance observations.

Keep three distinctions in view:

Observed evidence says what the response or rendered page contained.
A finding applies a documented rule or heuristic to that evidence.
An action depends on the page’s purpose, intended canonical, template, and business context.

A different canonical is not automatically wrong. Google treats a declared canonical as one signal and may select another URL. Its canonical URL guidance is the useful reference when a crawl finds conflicting signals.

Structured data is similar. Valid markup can make a page eligible for a search appearance, but Google does not guarantee that appearance. Check the visible page, required properties, and the relevant structured data policies before opening a ticket.

Render JavaScript when the source response is not enough

seo crawl https://example.com --rendering on --max-pages 50

Use --rendering on when the links, canonicals, or main content are missing from the initial HTML. auto is the default. It only opens a browser when the response looks client-rendered. off keeps the raw HTTP response.

The crawler uses a local Chrome or Chromium browser. It reuses a browser process during a crawl and keeps each page in its own browser context. If no compatible browser is available, the report says that rendered evidence is unavailable and keeps the raw response separate. It does not pretend raw HTML was rendered.

Install Chrome or Chromium in the usual way. For a Playwright-managed browser, run the exact command shown by the CLI. --js remains an alias for --rendering on while existing scripts move to the explicit mode.

When a browser renders a page, JSON output includes fetchDiagnostics.rendering.documentDifference. It shows whether the title, description, canonical, robots directives, headings, links, visible content, or structured data changed from the raw response. Saved crawls keep compact snapshots and text or link fingerprints rather than a second complete DOM.

Google’s JavaScript SEO guidance explains how rendered HTML affects discovery and indexing. A local rendered crawl is still your observation, not a copy of Google’s renderer or index.

Save a baseline and compare the next release

seo crawl --project example --save
seo crawl-reports --project example
seo crawl-reports --project example --compare latest --against previous

The comparison reports pages and findings that were added, removed, fixed, or changed. Check the crawl limits and URL sets on both sides before calling a difference a regression. Comparing a 100-page sample with a 500-page crawl can produce dramatic numbers for a very boring reason.

Run a fresh crawl after the deployment, then use the affected URLs to verify the template or route that changed.

Export the part another person needs

seo crawl https://example.com --format json --output crawl.json
seo crawl https://example.com --format csv --csv issues --output issues.csv
seo crawl https://example.com --format html --output report.html
seo crawl https://example.com --format markdown --output tickets.md
seo crawl --sitemap-url https://example.com/sitemap.xml --health --format junit --output sitemap-health.xml

JSON preserves the structured contract for scripts and agents. CSV works for sorting issues or pages. HTML gives you a local standalone report, while Markdown is useful for a limited implementation brief. JUnit creates one test case per sitemap document and listed URL with status, redirect, network, robots, and access-block evidence.

Fail CI on a severity you have agreed to enforce

seo crawl --sitemap-url https://example.com/sitemap.xml --health --format junit --output sitemap-health.xml --fail-on high

The process exits unsuccessfully when findings meet the selected severity or higher. JSON output still includes the triggering evidence.

A CI gate is strongest when the crawl scope is stable and the team agrees what that severity means. Save the output as an artifact. Without the affected URLs and rule evidence, a red build is just a red build.

Follow a failed health URL with a focused page audit or full crawl. Do not make every deployment download and analyse the whole site.

Use the CLI automation guide for explicit JSON runs. The AI search evidence guide covers technical eligibility checks that reuse a saved crawl.