---
title: "robots.txt validator and tester | SEO Skill"
description: "Fetch or paste a robots.txt file, find ignored lines, and test the exact rule that allows or blocks a URL for search and AI crawlers."
canonical: "https://seoskill.dev/tools/robots-txt-validator"
language: "en"
---

# robots.txt validator and tester

Fetch a live robots.txt file or test custom rules. See which user-agent group applies, the winning rule, and its source line for up to 100 URLs.

**Rules parsed**

Groups

0

Rules

0

Errors

0

Warnings

0

URLs tested

0

### Source details

Requested

Final source

Response

File

Redirects

### Copy or download the report

JSON includes the source, parsed groups, line findings, limits, and every URL verdict.

### URL test results

A result covers robots.txt matching only. It does not test a firewall, login, page response, or indexing state.

| URL | Result | Winning rule | Reason |
| --- | --- | --- | --- |

URL tests were not run because the live response did not establish which rules currently apply.

### Line findings

Parseable rules remain active even when another line is ignored.

| Level | Line | Finding | Details |
| --- | --- | --- | --- |

No line-level problems were found in the parsed content.

### Parsed groups and sitemaps

Repeated groups for the same specific crawler are combined when a URL is tested.

#### Declared sitemaps

## Test live rules or a draft file

Live mode requests the robots.txt file at the site's root. Custom mode checks pasted rules inside your browser, which is useful before a change reaches production. Both modes use the same parser and URL matching rules.

The result keeps parseable rules even when another line is ignored. That matters because crawlers do not normally discard a whole file over one bad line.

## How the winning rule is selected

The tester follows the [Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html). It selects the most specific user-agent group, matches paths with case sensitivity, supports `*` and terminal `$`, and uses the rule that matches the most URL octets. An Allow rule wins when equally specific Allow and Disallow rules conflict.

Repeated groups for the same specific crawler are combined. A specific group is not combined with the global `User-agent: *` group. Google's [robots.txt parser documentation](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) shows the same grouping and matching behavior.

## Allowed does not prove a crawler can fetch the page

An allowed result means no matching robots.txt rule blocks the URL. A firewall, login, bot challenge, network failure, or page response can still stop a request. The tool does not impersonate a crawler or verify its source IP address.

robots.txt is also a crawl control, not access control. Google may show a blocked URL without a snippet when it discovers that address elsewhere. Use authentication for private content and a crawlable `noindex` directive when the goal is to keep a public page out of Google Search.

## Missing files and server errors mean different things

Google generally treats a missing robots.txt file and most other 4xx responses as no crawl restrictions. A 429 response, server error, DNS failure, or timeout is different because a cached copy may continue to apply. The checker reports that state as uncertain rather than guessing which rules a crawler currently holds.

## Search and AI crawlers use separate tokens

The presets separate search crawlers from model training controls. OpenAI documents `OAI-SearchBot` for ChatGPT search and `GPTBot` for potential training. Anthropic documents separate Claude search and training crawlers. Blocking one does not describe the access of the other.

Read the current crawler guidance from [OpenAI](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq), [Anthropic](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), and [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) before making a policy change. Crawler names and uses can change.