Crawl a site with one command
The crawler is part of the seo CLI. You do not need to connect Google or save a key to crawl, so once the package is installed you can point it at any site:
npm i -g seo
seo crawl https://example.comOr ask your agent to crawl the site and tell you what to fix first. The crawl fetches a lot of pages, so it always runs on your machine. If your agent is connected to the hosted MCP, it is told to use the CLI for the crawl.
Crawl a local or staging build
Anything your machine can reach can be crawled, including a dev server. Start your site locally and crawl it before the release goes out:
seo crawl http://localhost:4321Staging sites often block every crawler in robots.txt. The crawler respects robots.txt by default, so it finds almost nothing there. Add --no-robots to crawl it anyway. The crawler sends no password or cookie, so a staging site behind a login only shows it the login page.
To stop a release when the crawl finds serious problems, add --fail-on high. The command then exits with an error when there is a high severity issue, which a CI step or a deploy script can check.
What the crawler checks
| Area | What it reports |
|---|---|
| Status codes | Pages that return errors, and pages that redirect. |
| Internal links | Links on your pages that land on a broken URL. |
| Titles and headings | Missing or duplicate titles and meta descriptions, and pages with no H1. |
| Indexing tags | Canonical tags, robots meta tags and noindex, as each page serves them. |
| Page basics | The viewport tag, the page language and Open Graph tags for link previews. |
| AI agents | Which AI bots your robots.txt allows, whether /llms.txt works, and whether pages have a Markdown copy. |
Read the results
Fixes come first, ranked by severity, each with the URLs that show the problem. Below them is a list to review before changing anything. A noindex tag or a canonical pointing elsewhere is reported as what the page says, because it is only a problem if you did not mean it.
The crawl stops at 500 pages unless you raise --max-pages (up to 10,000). For a big site, start with a quick pass over the sitemap that checks status codes and redirects without reading every page:
seo crawl --sitemap-url https://example.com/sitemap.xml --healthKeep or share a crawl
Nothing is saved for you. Write the result to a file when you want to keep it or hand it to someone: an HTML report, a CSV of issues or pages, Markdown tickets, or JSON for your agent.
seo crawl https://example.com --format html --output crawl.htmlPages are cached on your machine between crawls. After a deploy, add --refresh so every page is fetched again.
From a crawl to an SEO audit
A crawl shows what your pages say. It cannot tell you what Google did with them, or which issues cost you clicks. The SEO audit tool puts the crawl next to Google's index status and your Search Console data, and the Google index checker asks Google about each page directly.
The crawl on your machine is free, as often as you like. If you would rather not run it yourself, the weekly site audit ($50 a month per site) crawls the site every week on the hosted server and emails you what changed.