SEO log file analyzer

See which crawlers request your pages

The analyzer groups requests by crawler family, URL path, and HTTP status. Query strings are removed so tracking parameters do not split one page into hundreds of rows. It also shows the first and last valid request dates found in the file.

Combined Apache and NGINX access logs work directly. JSONL and NDJSON exports can use common names for the timestamp, method, URL, status, response bytes, and user agent.

Your raw access log stays on your machine

The page does not upload the log or keep a searchable copy of its raw requests. Large files can be cancelled by closing the tab or choosing another file.

User-agent matches need verification

A matching user-agent string shows what the request claimed to be. It does not prove that Google, Bing, OpenAI, Anthropic, or another company sent the request. Check important findings against the original log and your server configuration before blocking or allowing a crawler.

A partial result means the file contained malformed rows or reached a processing limit. Treat missing crawler activity as unknown in that case, rather than a clean zero.

Read the crawler summary

Start with the status codes for search crawlers. Mostly 200s is healthy. A run of 5xx responses tells Googlebot your server is struggling, and Google slows its crawling when that happens. Lots of 404s for old URLs are normal and harmless unless they are pages that should exist. Lots of 301s mean your internal links or sitemap still point at old addresses.

Then look at the paths. Crawlers spending most of their requests on filters, search pages or old parameters, while new pages get few visits, is a sign the site wastes crawling on pages that do not matter.

Why log files matter for SEO

Search Console shows a sample of what Google crawled. Your access log shows every request, so it is the only place to see exactly which pages crawlers fetch, how often and what your server sent back. That matters most for large sites, where Google cannot crawl everything often, and after a migration, when you need to see crawlers reach the new URLs.

For a small site, crawling is rarely the problem. If Googlebot fetches your pages and gets 200s, look elsewhere.

Verify a crawler before you act

Anyone can send a request that calls itself Googlebot. To confirm a real Google crawler, do a reverse DNS lookup on the IP address: it should resolve to googlebot.com, google.com or googleusercontent.com, and a forward lookup on that name should return the same IP. Google also publishes its crawler IP ranges. Do the same for other crawlers with their own published details before you block or allow them.

Use the results with your AI agent

SEO Skill has no log command, and this page keeps your log in the browser. Download the CSV summary from here and give it to your agent. Next to a seo crawl of the same site, it can find pages crawlers keep requesting that your site no longer links to, and linked pages crawlers never reach. Ask:

Here is the crawler CSV from my server logs and a crawl of example.com. Which pages does Googlebot request that the site no longer links to?

Add the hosted MCP server to your agent and sign in with Google:

Log file analysis questions