---
title: "Server log crawler analysis | SEO Skill"
description: "Stream a local combined or JSONL access log to review observed search and AI crawler requests, response errors and paths without retaining raw events."
canonical: "https://seoskill.dev/docs/reports/server-log-analysis"
language: "en"
---

# Server log crawler analysis

See which known crawler user agents reached the server and where they met redirects or errors. The report aggregates the file as it reads, keeps memory and output bounded, and shows when limits or malformed rows reduced coverage.

## Install and run the report

Install the CLI once, then run `seo start` to connect your site and Google data. After setup you can run this report yourself or ask your agent to run it.

Install the SEO CLI globally.

```
npm i -g seo
```

Run setup after installation.

```
seo start
```

Run review crawler activity in server logs through the report catalog.

```
seo reports run server-log-analysis --params '{"file":"./access.log","format":"combined","rowLimit":1000000,"pathLimit":25000,"limit":100}' --json
```

## What you get from this report

Stream a local access log into bounded crawler, status, and path evidence without building another raw log database.

Best for

You have a combined or JSONL access log and need evidence of real server requests.

Needs

Combined or JSONL access log

You get

Observed crawler request totals with 2xx, 3xx, 4xx, and 5xx breakdowns and latest timestamps.

Use next

Sitemap health check and technical site crawl

## What you need before you run it

This report uses method, path, timestamp, response status, response bytes, and user agent parsed from an explicit local file.

### Combined or JSONL access log

Provides timestamps, request paths, response statuses, bytes, and user-agent strings from the selected local file.

### Explicit work and output limits

Bounds input bytes, rows, line length, unique crawler-path aggregates, and returned detail.

## What the monitor checks

The monitor compares limited snapshots over time. It keeps failed, deferred and unchecked pages separate so missing evidence does not look like a recovery.

01

Parses common combined log fields or supported JSON aliases and records malformed rows separately.

02

Groups a narrow list of known search and AI crawler user-agent tokens, response classes, and normalized paths without retaining raw requests.

03

Keeps full-file, row, byte, path, parser, and output coverage visible beside the aggregates.

## Check what ran before trusting the status

Start with the selected pages, dates and quota use. A page that was not checked has an unknown status, even when the rest of the run looks healthy.

| Read first | Why it matters |
| --- | --- |
| Result | Observed crawler request totals with 2xx, 3xx, 4xx, and 5xx breakdowns and latest timestamps. |
| More detail | A bounded list of crawler and path aggregates, file provenance, warnings, and explicit omissions. |
| Read it with | Start with file coverage and warnings. Investigate repeated crawler 4xx or 5xx paths against the original log and current server behavior. |

## What the result cannot prove

User-agent strings can be absent or spoofed, and the supplied file may omit servers, requests, or time periods. Path rankings cover retained aggregates only when the path limit is reached.

Start with file coverage and warnings. Investigate repeated crawler 4xx or 5xx paths against the original log and current server behavior. Check deployment, firewall, CDN, and server configuration for repeated crawler errors.

## Use a different report for these jobs

You need verified crawler identity from reverse DNS or source IP validation. You need durable raw-event storage and arbitrary log search.

[

### You need to check the current response and page controls for a path from the log.

Audit the live URL after reviewing the source log evidence. The live check can inspect the current response and page controls, but it cannot recreate the historical server request.

SEO page audit

](https://seoskill.dev/docs/reports/audit-page)

### You need durable raw-event storage, arbitrary searches, or identity verification from source networks.

Use the server or a dedicated log platform for that job. This report deliberately performs one bounded local aggregation and does not retain raw events or verify crawler source IPs.

Needs more evidence

## Use the report with an agent or in code

The command above is enough for most work. These options let an agent, script or application run the same report with explicit input and structured output.

### CLI

Use the report catalog when an agent, script or CI job needs predictable JSON instead of the normal terminal output. Check the current input first, then run the report with the validated parameters.

Check the current input before the agent or script runs the report.

```
seo reports describe server-log-analysis --json
```

Once the input is valid, run the report through the report catalog.

```
seo reports run server-log-analysis --params '{"file":"./access.log","format":"combined","rowLimit":1000000,"pathLimit":25000,"limit":100}' --json
```

### MCP

An MCP agent follows the same two-step flow. It inspects the current schema, fills the required fields and only then runs the report.

Ask the MCP server for the current report input.

```
{
  "id": "server-log-analysis"
}
```

Run the report after the agent has checked and filled the input.

```
{
  "id": "server-log-analysis",
  "params": {
    "file": "./access.log",
    "format": "combined",
    "rowLimit": 1000000,
    "pathLimit": 25000,
    "limit": 100
  }
}
```

### TypeScript

Install the package in a Node 22 project when your application needs to run the report directly. The result includes the same evidence, warnings and limits returned through the CLI and MCP server.

Add the SEO package to your project.

```
npm install seo
```

Run the report from your TypeScript application.

```
import { executeReport } from 'seo/mcp'

const result = await executeReport(
  'server-log-analysis',
  {
  "file": "./access.log",
  "format": "combined",
  "rowLimit": 1000000,
  "pathLimit": 25000,
  "limit": 100
},
)

console.log(result)
```

## What to do next

Check important crawler errors against the original log and current server configuration. Confirm crawler identity separately when the decision depends on more than the observed user-agent string.

1.  01Trace an important redirecting or failing path against the live site.
2.  02Check deployment, firewall, CDN, and server configuration for repeated crawler errors.

## Related reports

[

Check sitemap URL health first, then run a full technical crawl only when page-level evidence is needed.

Sitemap health check and technical site crawl

](https://seoskill.dev/docs/reports/site-crawl)[

Follow every hop from one URL to its final page and catch loops, long chains, broken destinations, or conflicting index signals.

Redirect trace

](https://seoskill.dev/docs/reports/redirect-trace)[

See every technical check built into the local crawler and find the right rule for a focused follow-up.

SEO crawler rules

](https://seoskill.dev/docs/reports/crawler-rules)

## Sources used by this report

These primary sources define the provider data or search controls used by the result.

-   [NGINX access log format reference](https://nginx.org/en/docs/http/ngx_http_log_module.html)