---
title: "Free SEO crawler for any site | SEO Skill"
description: "A free SEO crawler that runs on your machine with no sign-in. Crawl any site, including localhost and staging, for broken links, redirects, titles, canonicals and noindex tags."
canonical: "https://seoskill.dev/features/seo-crawler"
language: "en"
---

# Free SEO crawler for any site

Crawl a site from your own machine and get a ranked list of broken pages, redirects, missing titles and stray noindex tags. It is free, needs no account, and works on a local or staging build before you ship.

Run it in a terminal, then sign in with Google.

Or run it on your machine

Installs the CLI, signs you in and adds the skill to your agents. Site crawls and Bing data run this way, on your own machine. A crawl fetches many pages, and Bing blocks our hosted server.

## Crawl a site with one command

The crawler is part of the `seo` CLI. You do not need to connect Google or save a key to crawl, so once the package is installed you can point it at any site:

```
npm i -g seo
seo crawl https://example.com
```

Or ask your agent to crawl the site and tell you what to fix first. The crawl fetches a lot of pages, so it always runs on your machine. If your agent is connected to the hosted MCP, it is told to use the CLI for the crawl.

## Crawl a local or staging build

Anything your machine can reach can be crawled, including a dev server. Start your site locally and crawl it before the release goes out:

```
seo crawl http://localhost:4321
```

Staging sites often block every crawler in robots.txt. The crawler respects robots.txt by default, so it finds almost nothing there. Add `--no-robots` to crawl it anyway. The crawler sends no password or cookie, so a staging site behind a login only shows it the login page.

To stop a release when the crawl finds serious problems, add `--fail-on high`. The command then exits with an error when there is a high severity issue, which a CI step or a deploy script can check.

## What the crawler checks

| Area | What it reports |
| --- | --- |
| Status codes | Pages that return errors, and pages that redirect. |
| Internal links | Links on your pages that land on a broken URL. |
| Titles and headings | Missing or duplicate titles and meta descriptions, and pages with no H1. |
| Indexing tags | Canonical tags, robots meta tags and noindex, as each page serves them. |
| Page basics | The viewport tag, the page language and Open Graph tags for link previews. |
| AI agents | Which AI bots your robots.txt allows, whether `/llms.txt` works, and whether pages have a Markdown copy. |

## Read the results

Fixes come first, ranked by severity, each with the URLs that show the problem. Below them is a list to review before changing anything. A noindex tag or a canonical pointing elsewhere is reported as what the page says, because it is only a problem if you did not mean it.

The crawl stops at 500 pages unless you raise `--max-pages` (up to 10,000). For a big site, start with a quick pass over the sitemap that checks status codes and redirects without reading every page:

```
seo crawl --sitemap-url https://example.com/sitemap.xml --health
```

## Keep or share a crawl

Nothing is saved for you. Write the result to a file when you want to keep it or hand it to someone: an HTML report, a CSV of issues or pages, Markdown tickets, or JSON for your agent.

```
seo crawl https://example.com --format html --output crawl.html
```

Pages are cached on your machine between crawls. After a deploy, add `--refresh` so every page is fetched again.

## From a crawl to an SEO audit

A crawl shows what your pages say. It cannot tell you what Google did with them, or which issues cost you clicks. The [SEO audit tool](https://seoskill.dev/features/seo-audit) puts the crawl next to Google's index status and your Search Console data, and the [Google index checker](https://seoskill.dev/features/google-index-checker) asks Google about each page directly.

The crawl on your machine is free, as often as you like. If you would rather not run it yourself, the weekly site audit ($50 a month per site) crawls the site every week on the hosted server and emails you what changed.

## Read the docs

[

### seo crawl

Every crawl option: modes, page limits, output formats and the AI agent checks.

seo crawl docs](https://seoskill.dev/docs/crawl)[

### seo internal-links

Pages that need more internal links, and where to add them.

seo internal-links docs](https://seoskill.dev/docs/internal-links)[

### seo page

One URL: what it serves now, its Google index status and its queries.

seo page docs](https://seoskill.dev/docs/page)[

### CLI

Install the CLI and run any command from your terminal.

CLI docs](https://seoskill.dev/docs/cli)