robots.txt validator and tester
Fetch a live robots.txt file or test custom rules. See which user-agent group applies, the winning rule, and its source line for up to 100 URLs.
URL test results
A result covers robots.txt matching only. It does not test a firewall, login, page response, or indexing state.
| URL | Result | Winning rule | Reason |
|---|
URL tests were not run because the live response did not establish which rules currently apply.
Line findings
Parseable rules remain active even when another line is ignored.
| Level | Line | Finding | Details |
|---|
No line-level problems were found in the parsed content.
Parsed groups and sitemaps
Repeated groups for the same specific crawler are combined when a URL is tested.
Declared sitemaps
Test live rules or a draft file
Live mode requests the robots.txt file at the site's root. Custom mode checks pasted rules inside your browser, which is useful before a change reaches production. Both modes use the same parser and URL matching rules.
The result keeps parseable rules even when another line is ignored. That matters because crawlers do not normally discard a whole file over one bad line.
How the winning rule is selected
The tester follows the Robots Exclusion Protocol. It selects the most specific user-agent group, matches paths with case sensitivity, supports * and terminal $, and uses the rule that matches the most URL octets. An Allow rule wins when equally specific Allow and Disallow rules conflict.
Repeated groups for the same specific crawler are combined. A specific group is not combined with the global User-agent: * group. Google's robots.txt parser documentation shows the same grouping and matching behavior.
Allowed does not prove a crawler can fetch the page
An allowed result means no matching robots.txt rule blocks the URL. A firewall, login, bot challenge, network failure, or page response can still stop a request. The tool does not impersonate a crawler or verify its source IP address.
robots.txt is also a crawl control, not access control. Google may show a blocked URL without a snippet when it discovers that address elsewhere. Use authentication for private content and a crawlable noindex directive when the goal is to keep a public page out of Google Search.
Missing files and server errors mean different things
Google generally treats a missing robots.txt file and most other 4xx responses as no crawl restrictions. A 429 response, server error, DNS failure, or timeout is different because a cached copy may continue to apply. The checker reports that state as uncertain rather than guessing which rules a crawler currently holds.
Search and AI crawlers use separate tokens
The presets separate search crawlers from model training controls. OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for potential training. Anthropic documents separate Claude search and training crawlers. Blocking one does not describe the access of the other.
Read the current crawler guidance from OpenAI, Anthropic, and Perplexity before making a policy change. Crawler names and uses can change.