Server log crawler analysis
See which known crawler user agents reached the server and where they met redirects or errors. The report aggregates the file as it reads, keeps memory and output bounded, and shows when limits or malformed rows reduced coverage.
Install and run the report
Install the CLI once, then run seo start to connect your site and Google data. After setup you can run this report yourself or ask your agent to run it.
Runafter install
Install the SEO CLI globally.
npm i -g seoRun setup after installation.
seo startRun review crawler activity in server logs through the report catalog.
seo reports run server-log-analysis --params '{"file":"./access.log","format":"combined","rowLimit":1000000,"pathLimit":25000,"limit":100}' --jsonWhat you get from this report
Stream a local access log into bounded crawler, status, and path evidence without building another raw log database.
- Best for
- You have a combined or JSONL access log and need evidence of real server requests.
- Needs
- Combined or JSONL access log
- You get
- Observed crawler request totals with 2xx, 3xx, 4xx, and 5xx breakdowns and latest timestamps.
- Use next
- Sitemap health check and technical site crawl
What you need before you run it
This report uses method, path, timestamp, response status, response bytes, and user agent parsed from an explicit local file.
Combined or JSONL access log
Provides timestamps, request paths, response statuses, bytes, and user-agent strings from the selected local file.
Explicit work and output limits
Bounds input bytes, rows, line length, unique crawler-path aggregates, and returned detail.
What the monitor checks
The monitor compares limited snapshots over time. It keeps failed, deferred and unchecked pages separate so missing evidence does not look like a recovery.
01
Parses common combined log fields or supported JSON aliases and records malformed rows separately.
02
Groups a narrow list of known search and AI crawler user-agent tokens, response classes, and normalized paths without retaining raw requests.
03
Keeps full-file, row, byte, path, parser, and output coverage visible beside the aggregates.
Check what ran before trusting the status
Start with the selected pages, dates and quota use. A page that was not checked has an unknown status, even when the rest of the run looks healthy.
| Read first | Why it matters |
|---|---|
| Result | Observed crawler request totals with 2xx, 3xx, 4xx, and 5xx breakdowns and latest timestamps. |
| More detail | A bounded list of crawler and path aggregates, file provenance, warnings, and explicit omissions. |
| Read it with | Start with file coverage and warnings. Investigate repeated crawler 4xx or 5xx paths against the original log and current server behavior. |
What the result cannot prove
User-agent strings can be absent or spoofed, and the supplied file may omit servers, requests, or time periods. Path rankings cover retained aggregates only when the path limit is reached.
Start with file coverage and warnings. Investigate repeated crawler 4xx or 5xx paths against the original log and current server behavior. Check deployment, firewall, CDN, and server configuration for repeated crawler errors.
Use a different report for these jobs
You need verified crawler identity from reverse DNS or source IP validation. You need durable raw-event storage and arbitrary log search.
Use the report with an agent or in code
The command above is enough for most work. These options let an agent, script or application run the same report with explicit input and structured output.
CLI
Use the report catalog when an agent, script or CI job needs predictable JSON instead of the normal terminal output. Check the current input first, then run the report with the validated parameters.
Check the current input before the agent or script runs the report.
seo reports describe server-log-analysis --jsonOnce the input is valid, run the report through the report catalog.
seo reports run server-log-analysis --params '{"file":"./access.log","format":"combined","rowLimit":1000000,"pathLimit":25000,"limit":100}' --jsonMCP
An MCP agent follows the same two-step flow. It inspects the current schema, fills the required fields and only then runs the report.
Ask the MCP server for the current report input.
{
"id": "server-log-analysis"
}Run the report after the agent has checked and filled the input.
{
"id": "server-log-analysis",
"params": {
"file": "./access.log",
"format": "combined",
"rowLimit": 1000000,
"pathLimit": 25000,
"limit": 100
}
}TypeScript
Install the package in a Node 22 project when your application needs to run the report directly. The result includes the same evidence, warnings and limits returned through the CLI and MCP server.
Add the SEO package to your project.
npm install seoRun the report from your TypeScript application.
import { executeReport } from 'seo/mcp'
const result = await executeReport(
'server-log-analysis',
{
"file": "./access.log",
"format": "combined",
"rowLimit": 1000000,
"pathLimit": 25000,
"limit": 100
},
)
console.log(result)What to do next
Check important crawler errors against the original log and current server configuration. Confirm crawler identity separately when the decision depends on more than the observed user-agent string.
- 01Trace an important redirecting or failing path against the live site.
- 02Check deployment, firewall, CDN, and server configuration for repeated crawler errors.
Related reports
Check sitemap URL health first, then run a full technical crawl only when page-level evidence is needed.
Follow every hop from one URL to its final page and catch loops, long chains, broken destinations, or conflicting index signals.
See every technical check built into the local crawler and find the right rule for a focused follow-up.
Sources used by this report
These primary sources define the provider data or search controls used by the result.