Server log crawler analysis

See which known crawler user agents reached the server and where they met redirects or errors. The report aggregates the file as it reads, keeps memory and output limited, and shows when limits or malformed rows reduced coverage.

Install and run the report

Install the CLI once, then run seo start to connect your site and Google data. After setup you can run this report yourself or ask your agent to run it.

Runafter install

Install the SEO CLI globally.

npm i -g seo

Run setup after installation.

seo start

Run review crawler activity in server logs through the report catalog.

seo reports run server-log-analysis --params '{"file":"./access.log","format":"combined","rowLimit":1000000,"pathLimit":25000,"limit":100}' --json

What you get from this report

Stream a local combined or JSONL access log to review observed search and AI crawler requests, response errors and paths without retaining raw events.

Best for: You have a combined or JSONL access log and need evidence of real server requests.
Needs: Combined or JSONL access log
You get: Stream a local combined or JSONL access log to review observed search and AI crawler requests, response errors and paths without retaining raw events.
Use next: Sitemap health check and technical site crawl

Every report follows the same rules for missing data and outside estimates.See how report data works.

What you need before you run it

The report uses the information below. Optional sources are only used when you ask for them.

Combined or JSONL access log

Provides timestamps, request paths, response statuses, bytes, and user-agent strings from the selected local file.

Explicit work and output limits

Bounds input bytes, rows, line length, unique crawler-path aggregates, and returned detail.

Use a different report for these jobs

These jobs need a different report or a manual review.

You want to inspect one access log in the browser and download CSV files.

Use the browser tool for a one-off local check. The raw log stays in the tab and the same parser produces crawler, path, error, and status-code CSV files.

Open the SEO log file analyzer

You need to check the current response and page controls for a path from the log.

Audit the live URL after reviewing the source log evidence. The live check can inspect the current response and page controls, but it cannot recreate the historical server request.

SEO page audit

You need durable raw-event storage, arbitrary searches, or identity verification from source networks.

Use the server or a dedicated log platform for that job. This report deliberately performs one limited local aggregation and does not retain raw events or verify crawler source IPs.

Needs more evidence

Use the report with an agent or in code

The command above is enough for most work. These options let an agent, script or application run the same report with explicit input and structured output.

Use the report catalog when an agent, script or CI job needs predictable JSON instead of the normal terminal output. Check the current input first, then run the report with the validated parameters.

Check the current input before the agent or script runs the report.

seo reports describe server-log-analysis --json

Once the input is valid, run the report through the report catalog.

seo reports run server-log-analysis --params '{"file":"./access.log","format":"combined","rowLimit":1000000,"pathLimit":25000,"limit":100}' --json

An MCP agent follows the same two-step flow. It inspects the current schema, fills the required fields and only then runs the report.

Ask the MCP server for the current report input.

{
  "id": "server-log-analysis"
}

Run the report after the agent has checked and filled the input.

{
  "id": "server-log-analysis",
  "params": {
    "file": "./access.log",
    "format": "combined",
    "rowLimit": 1000000,
    "pathLimit": 25000,
    "limit": 100
  }
}

Install the package in a Node 22 project when your application needs to run the report directly. The result includes the same evidence, warnings and limits returned through the CLI and MCP server.

Add the SEO package to your project.

npm install seo

Run the report from your TypeScript application.

import { executeReport } from 'seo/mcp'

const result = await executeReport(
  'server-log-analysis',
  {
  "file": "./access.log",
  "format": "combined",
  "rowLimit": 1000000,
  "pathLimit": 25000,
  "limit": 100
},
)

console.log(result)

What to do next

Check important crawler errors against the original log and current server configuration. Confirm crawler identity separately when the decision depends on more than the observed user-agent string.

01Trace an important redirecting or failing path against the live site.
02Check deployment, firewall, CDN, and server configuration for repeated crawler errors.

Check sitemap URL status, redirects and access first. Run the full technical SEO crawl only when you need page content, links, metadata and affected URLs.

Sitemap health check and technical site crawl

Follow every redirect hop from one URL to its final page and check status codes, loops, chains, canonicals and link safety before updating links.

Redirect trace

Browse the technical checks built into the SEO crawler and find the exact rule ID needed for plain-English guidance or affected URLs from a saved crawl.

SEO crawler rules

Sources used by this report

These primary sources define the provider data or search controls used by the result.

NGINX access log format reference