Crawler Log Analyser
Crawler Log Analyser is a free, open-source Linux tool for operators of self-hosted static sites running nginx.
What Crawler Log Analyser is
Crawler Log Analyser is a free, open-source Linux tool for operators of self-hosted static sites running nginx. It reads combined or extended seo_crawl access logs, including rotated .gz files, then separates search and AI crawlers. From observed responses, it scores Googlebot and wider search-crawler health. It can expose deployment-related problems such as robots.txt failures, bursts of content 404 responses and crawl gaps after 404s. Additional checks identify sitemap-path user-agent spoofing and changing bot identity claims from one IP address. Reports are available as text, Markdown or JSON, and critical findings can produce a non-zero exit code for cron and CI workflows. The single-file Python script has no telemetry, dashboard, database, external service or external dependency. A TOML file can set ignored IPs, launch dates and an AI crawler discovery window.
Good
- Parses rotated .gz nginx logs
- Classifies search and AI crawlers
- Detects deployment-window crawler anomalies
- Produces text, Markdown or JSON reports
- Supports cron and CI exit codes
Watch out
- Requires Python 3.11 or later
- No stated Apache or IIS support guarantee
- No Slack, Prometheus, Elasticsearch or SaaS integrations
- Feature work frozen through 31 August 2026
Who it suits
It suits operators of self-hosted static sites who inspect real crawler behaviour in nginx logs. Linux users comfortable with Python-based, self-hosted tools are the intended audience.
Verdict
Crawler Log Analyser provides focused crawler observability without external services or telemetry. Its nginx and Python requirements, plus the feature freeze, define the operational trade-offs.
The facts
Each one read from Crawler Log Analyser's own pages, with the date it was checked.
What it does
- Additional detections
- It can detect sitemap-path user-agent spoofing and rotating bot identity claims from a single IP.checked 4 Oct 2026
- Anomaly detection
- It detects crawler-visible deploy-window issues including robots.txt failures, content-404 bursts, and post-404 crawl gaps.checked 10 Oct 2026
- Configuration
- A TOML config file can specify ignored source IPs, site launch dates, and an AI crawler discovery window.checked 10 Oct 2026
- Crawler analysis
- It classifies search and AI crawlers and scores Googlebot and broader search-crawler health from observed responses.checked 4 Oct 2026
- Crawler insights
- It classifies search and AI crawlers and scores Googlebot crawl health from observed responses.checked 10 Oct 2026
- Data handling
- The README says the tool has no database, telemetry, or external services.checked 10 Oct 2026
- Deploy anomalies
- It detects crawler-visible deploy-window issues such as robots.txt failures, content-404 bursts, and post-404 crawl gaps.checked 4 Oct 2026
- Feature status
- The feature-freeze page says no new feature work was planned through 31 August 2026 and that bug fixes and security fixes remain welcome.checked 10 Oct 2026
- Integrations
- The README says JSON output is the integration boundary and lists no Slack, Prometheus, Elasticsearch, or SaaS integrations.checked 10 Oct 2026
- Intended users
- The maker positions it for operators of self-hosted static sites who need to inspect real crawler behavior in nginx access logs.checked 4 Oct 2026
- License
- The project is licensed under AGPL-3.0-or-later; network services based on modified versions must make corresponding source code available under the AGPL.checked 10 Oct 2026
- Log analysis
- It parses nginx access logs, including rotated .gz logs, and supports combined and extended seo_crawl formats.checked 10 Oct 2026
- Log parsing
- It parses nginx access logs in combined or extended seo_crawl format, including rotated .gz logs.checked 4 Oct 2026
- Privacy and architecture
- The maker describes it as a single-file Python script with no telemetry, dashboards, database, external services, or external dependencies.checked 4 Oct 2026
- Project status
- The repository identifies v1.10.2 as its status and says feature work was frozen through 31 August 2026, while bug fixes and security fixes remain welcome.checked 4 Oct 2026
- Purpose
- The project describes itself as operational SEO observability for self-hosted static sites running nginx.checked 10 Oct 2026
- Reports
- It emits text, Markdown, or JSON reports and can exit non-zero on critical findings for cron and CI use.checked 4 Oct 2026
- Reports and automation
- It emits text, Markdown, or JSON reports and can exit non-zero on critical findings for cron and CI use.checked 10 Oct 2026
- Requirements
- It requires Python 3.11 or later and has no external dependencies.checked 10 Oct 2026
- Support
- The README welcomes pull requests and says project review may be slow because it is maintained as time permits.checked 4 Oct 2026
- Support and maintenance
- The maintainer says review may be slow because the project is maintained as time permits.checked 10 Oct 2026
The company
- Headquarters
- Inverness, Scottish Highlands, UKchecked 28 Sep 2026
- Maker
- SpeyTech identifies Spey Systems Ltd as the company behind its repositories and describes it as a Scottish software and systems engineering company based in Inverness.checked 10 Oct 2026
Alternatives to Crawler Log Analyser
- 1
Screaming Frog SEO Log File Analyser
Free planLinuxmacOSWindows7.6From $139 a year - 2
CrawlGenius Log Analyzer
Free planWeb6.7From €49 a month - 3
SEO Log Auditor
Free planWebLinuxmacOSWindows6.7Free plan - 4
SEOMER
Free planWeb6.7From $12 a month - 5
LogLens
Free planWebAPISelf-hosted6.6From $19 a month - 6
AuditLabs SEO Log File Analyzer
Free planWeb6.5Free plan