SEO Log File Analyzer

What it does

AI crawlers

It detects and classifies AI crawlers by vendor and purpose, including training, search, and user-fetch categories.checked 9 Oct 2026

Bot comparison

Compare Bots shows crawl footprints side by side and a top-URLs-by-bot coverage matrix.checked 9 Oct 2026

Bot comparisons

Compare Bots shows crawl footprints side by side and a URL-by-bot coverage matrix.checked 4 Oct 2026

Bot verification

It verifies supported search engine bots using reverse and forward DNS and checks some AI crawlers against vendors’ published IP ranges; results are cached in .cache/.checked 9 Oct 2026

Deployment

The README instructs users to clone the repository, install its Python requirements, and run the app with Streamlit at localhost:8501.checked 4 Oct 2026

Geolocation

IP-to-country lookup uses ip-api.com’s free batch endpoint, requires no API key, and is cached locally.checked 4 Oct 2026

Geolocation integration

IP geolocation uses ip-api.com’s free batch endpoint without an API key, with a stated limit of about 15 batch requests per minute; private IPs are skipped.checked 9 Oct 2026

License

The repository identifies its license as MIT.checked 9 Oct 2026

Log formats

It supports Apache/Nginx NCSA formats, Cloudflare Logpush JSON, JSON logs, IIS/W3C Extended logs, and a custom regex and field list.checked 9 Oct 2026

Metrics

Its dashboard reports bot activity, response codes, crawled URLs, directories, bandwidth, IPs, and geolocation.checked 9 Oct 2026

Network requirement

Bot verification and geolocation are optional and make network requests when enabled.checked 9 Oct 2026

Network use

Bot verification and geolocation are optional and make network requests; the README says these options run slower.checked 4 Oct 2026

Period comparison

Compare Periods compares two date ranges using event, bot, URL, and status deltas, percentage changes, and top movers.checked 9 Oct 2026

Period comparisons

Compare Periods shows event, bot, URL, and status changes across two date ranges, including percentage changes and top movers.checked 4 Oct 2026

Purpose

The project analyzes server access logs for SEO and displays crawl metrics in an interactive Streamlit dashboard.checked 9 Oct 2026

Sample data

The app includes a bundled sample log, and the repository documents a script for generating sample logs.checked 9 Oct 2026

Setup

The README instructs users to clone the repository, install its Python requirements, and run it with Streamlit; it opens locally at localhost:8501.checked 9 Oct 2026