Technical SEO audit: crawl your site and act on the evidence

The crawler is the base of a technical SEO audit. It fetches pages from your machine and records what a crawler can observe at that moment, giving you a technical baseline, an affected URL set, and a way to verify changes after a release.

Run a useful first crawl

Start with the public URL you expect search engines and users to reach:

seo crawl https://example.com --save

The default site crawl stops at 500 pages and a click depth of 16. It can seed URLs from sitemaps, follows internal links, and respects robots.txt by default. Saving the result lets focused reports reuse the page evidence without fetching the site again.

Use a project profile to join the crawl to available Search Console and GA4 evidence:

seo crawl --project example --max-pages 500 --save

The live catalog exposes 50 documented crawler rules. List their stable IDs or inspect one rule’s evidence and repair guidance:

seo rules
seo explain --rule missing_meta_description

Keep the crawl focused and fast

Bound the crawl to the question in front of you:

seo crawl https://example.com --max-pages 100 --max-depth 4
seo crawl https://example.com --include '/products/' --exclude '/account/'
seo crawl --mode sitemap https://example.com
seo crawl --mode page --url https://example.com/pricing
seo crawl --mode list --urls-file changed-urls.txt

site mode discovers internal URLs from the start page and sitemap. sitemap mode starts from a sitemap. page fetches one page, while list keeps the run to URLs you provide. When robots.txt declares same-origin sitemap files, the crawler tries those first and records which ones supplied URLs in JSON. If none return URLs, it also tries /sitemap.xml.

sitemapDiscovery.roots[].documents keeps the response evidence for each fetched sitemap: HTTP status, content type, compression, byte counts, XML root, and any warning. Gzip-compressed sitemaps work. Malformed XML, a non-sitemap root, an unsuitable response, or a file over the 50 MiB protocol limit stays visible as partial or unavailable evidence rather than disappearing from the result.

When a sitemap redirects, that document keeps both the requested URL and the final URL. The response status belongs to the final response. Treat a redirect as evidence to inspect, not proof that the sitemap is broken.

sitemapDiscovery.roots[].lastmods records supplied lastmod values as unverified metadata. It counts values that parse and keeps small samples of malformed or future dates so you can inspect the generating system. The crawler never uses them to order discovery or claim that a page changed. An index-level lastmod describes the sitemap file, not every page it lists. Google only uses lastmod when it is consistently and verifiably accurate in its sitemap guidance.

Google describes a sitemap as a discovery signal rather than an indexing guarantee in its sitemap guidance. Google limits each sitemap to 50 MB uncompressed or 50,000 URLs in its sitemap build guide. Treat sitemap membership the same way in a crawl report. A listed URL deserves inspection, but its presence does not prove that Google indexed it.

External-link checks stay bounded

External-link checks run against a bounded sample of the URLs retained by the crawl. The crawler spreads that sample across source pages, keeps each selected check on the page that linked to it, and records the limits, deferred URLs, and request failures in externalLinkVerification JSON. A 404 is evidence about the external target. It does not mean the page containing that link failed to crawl.

Robots.txt controls crawling, not indexing

The crawler skips disallowed URLs by default. That matches the site’s stated crawl policy and avoids surprising a site owner.

seo crawl https://example.com --no-robots

Use --no-robots only when you control the target and need to diagnose rules that block your own crawl. Google notes that a disallowed URL can still appear in search results when it is discovered elsewhere. Its robots.txt documentation explains why robots.txt should not be treated as a removal tool.

The crawler reports the rule it observed. It cannot know whether the rule is intentional. The same caution applies to noindex, canonicals, redirects, and snippet controls.

Read technical findings without inventing an SEO rule

The crawl records response status, redirects, robots directives, canonicals, titles, descriptions, headings, internal links, images, structured data, hreflang, viewport configuration, security headers, social previews, and available performance observations.

Keep three distinctions in view:

1. Observed evidence says what the response or rendered page contained.
2. A finding applies a documented rule or heuristic to that evidence.
3. An action depends on the page’s purpose, intended canonical, template, and business context.

A different canonical is not automatically wrong. Google treats a declared canonical as one signal and may select another URL. Its canonical URL guidance is the useful reference when a crawl finds conflicting signals.

Structured data is similar. Valid markup can make a page eligible for a search appearance, but Google does not guarantee that appearance. Check the visible page, required properties, and the relevant structured data policies before opening a ticket.

Render JavaScript when the source response is not enough

seo crawl https://example.com --rendering on --max-pages 50

Use --rendering on when the links, canonicals, or main content are missing from the initial HTML. auto is the default. It only opens a browser when the response looks client-rendered. off keeps the raw HTTP response.

The crawler uses a local Chrome or Chromium browser. It reuses a browser process during a crawl and keeps each page in its own browser context. If no compatible browser is available, the report says that rendered evidence is unavailable and keeps the raw response separate. It does not pretend raw HTML was rendered.

Install Chrome or Chromium in the usual way. For a Playwright-managed browser, run the exact command shown by the CLI. --js remains an alias for --rendering on while existing scripts move to the explicit mode.

When a browser renders a page, JSON output includes fetchDiagnostics.rendering.documentDifference. It shows whether the title, description, canonical, robots directives, headings, links, visible content, or structured data changed from the raw response. Saved crawls keep compact snapshots and text or link fingerprints rather than a second complete DOM.

Google’s JavaScript SEO guidance explains how rendered HTML affects discovery and indexing. A local rendered crawl is still your observation, not a copy of Google’s renderer or index.

Save a baseline and compare the next release

seo crawl --project example --save
seo crawl-reports --project example
seo crawl-reports --project example --compare latest --against previous

The comparison reports pages and findings that were added, removed, fixed, or changed. Check the crawl limits and URL sets on both sides before calling a difference a regression. Comparing a 100-page sample with a 500-page crawl can produce dramatic numbers for a very boring reason.

Run a fresh crawl after the deployment, then use the affected URLs to verify the template or route that changed.

Export the part another person needs

seo crawl https://example.com --format json --output crawl.json
seo crawl https://example.com --format csv --csv issues --output issues.csv
seo crawl https://example.com --format html --output report.html
seo crawl https://example.com --format markdown --output tickets.md

JSON preserves the structured contract for scripts and agents. CSV works for sorting issues or pages. HTML gives you a local standalone report, while Markdown is useful for a bounded implementation brief.

Fail CI on a severity you have agreed to enforce

seo crawl https://example.com --fail-on high --json

The process exits unsuccessfully when findings meet the selected severity or higher. JSON output still includes the triggering evidence.

A CI gate is strongest when the crawl scope is stable and the team agrees what that severity means. Save the output as an artifact. Without the affected URLs and rule evidence, a red build is just a red build.

Use the CLI automation guide for explicit JSON runs. The AI search evidence guide covers technical eligibility checks that reuse a saved crawl.