robots.txt validator and tester

Test live rules or a draft file

Live mode requests the robots.txt file at the site's root. Custom mode checks pasted rules inside your browser, which is useful before a change reaches production. Both modes use the same parser and URL matching rules.

The result keeps parseable rules even when another line is ignored. That matters because crawlers do not normally discard a whole file over one bad line.

How the winning rule is selected

The tester follows the Robots Exclusion Protocol. It selects the most specific user-agent group, matches paths with case sensitivity, supports * and terminal $, and uses the rule that matches the most URL octets. An Allow rule wins when equally specific Allow and Disallow rules conflict.

Repeated groups for the same specific crawler are combined. A specific group is not combined with the global User-agent: * group. Google's robots.txt parser documentation shows the same grouping and matching behavior.

Allowed does not prove a crawler can fetch the page

An allowed result means no matching robots.txt rule blocks the URL. A firewall, login, bot challenge, network failure, or page response can still stop a request. The tool does not impersonate a crawler or verify its source IP address.

robots.txt is also a crawl control, not access control. Google may show a blocked URL without a snippet when it discovers that address elsewhere. Use authentication for private content and a crawlable noindex directive when the goal is to keep a public page out of Google Search.

Missing files and server errors mean different things

Google generally treats a missing robots.txt file and most other 4xx responses as no crawl restrictions. A 429 response, server error, DNS failure, or timeout is different because a cached copy may continue to apply. The checker reports that state as uncertain rather than guessing which rules a crawler currently holds.

Search and AI crawlers use separate tokens

The presets separate search crawlers from model training controls. OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for potential training. Anthropic documents separate Claude search and training crawlers. Blocking one does not describe the access of the other.

Read the current crawler guidance from OpenAI, Anthropic, and Perplexity before making a policy change. Crawler names and uses can change.

Fix common robots.txt mistakes

• A leftover Disallow: /. A staging file copied to production blocks the whole site. Test the home page for Googlebot after every launch.
• Blocked CSS and JavaScript. Google renders pages like a browser. If it cannot fetch the files a page needs, it may see a broken page.
• Noindex in robots.txt. Google ignores it. Put the noindex on the page and leave the page crawlable.
• Blocking a page you want removed. The block hides the noindex tag, so the URL can stay in results. Allow the crawl until the page drops out.
• Wrong case or a missing slash. Paths are case sensitive, so /Admin does not block /admin. Test the exact URLs.

Check robots.txt across your site with your agent

The seo crawl command reads your robots.txt while it crawls your site on your own machine. It lists which AI search crawlers and which AI training crawlers your file allows, apart from each other, and reports the pages your rules stop its own crawler from fetching. Your agent can then change the file in your codebase and test the new rules here before you ship. Ask:

Crawl example.com and tell me which AI search and training crawlers my robots.txt blocks, and which linked pages robots.txt stops the crawl from fetching

The crawl runs from the CLI. Install it and run setup:

npm i -g seo && seo start

Or add the hosted MCP server for Search Console work in your agent:

robots.txt questions