Spider-Lens

Spider-Lens is a self-hosted log-analysis project for Apache and Nginx, pairing traffic and crawler dashboards with a sitemap crawler and configurable alerts. It best suits site operators who want those tools under their own deployment rather than a paid usage tier. Its breadth is appealing, but the standalone app has explicit runtime and log-access requirements, and its AI assistant depends on a separate Gemini API key.

Overview

Spider-Lens brings server-log monitoring and technical site crawling into one project. Its dashboard covers requests, human visits, bots, error rates, HTTP status trends, bot distribution, and 12-week trends; separate tools examine crawler activity, response times, and sitemap-linked pages. This makes it a practical fit for teams that want to connect what crawlers request with what their site exposes, rather than relying on a dashboard alone.

The project is free and self-hosted, with no paid plans or usage limits stated. That avoids a stated subscription tier, but it also means deployment and access to Apache or Nginx logs are part of the decision. The native WordPress plugin can avoid installing Node.js on the server; the standalone application cannot.

Key features

Log and crawler monitoring

Spider-Lens detects 16 or more bots and visualizes crawler volume and crawl budget. The dashboard’s status and error trends help put bot activity alongside broader traffic and HTTP health, while crawler segmentation can help distinguish patterns within the activity. The project is focused on server logs, so its value depends on having the relevant Apache or Nginx log files accessible.

Performance and site crawling

Per-URL Time to First Byte reporting, configurable slow thresholds, and CSV and Excel exports make the performance view useful for identifying and sharing slow-page findings. The sitemap crawler follows internal links and collects titles, H1 headings, word counts, canonicals, robots directives, and crawl depth. Together, these functions extend the tool beyond log review into a structured look at pages reachable through a sitemap.

Alerts, integrations, and AI

Configurable SMTP email and webhook alerts cover 404 spikes, server errors, missing Googlebot, and weekly reports. The project also documents Discord and Slack webhooks and an MCP server for Claude Code or other compatible clients. Nova uses Google Gemini and requires a Gemini API key; without one, the rest of the application continues to work, while Nova and AI Analysis show a configuration prompt. The README says Nova sends Google a compact JSON summary of about 500 tokens per request rather than raw log data.

Security and languages

JWT authentication uses a seven-day token. Spider-Lens advises changing the default JWT secret and admin password before deployment, an important setup step for a self-hosted application. Its interface supports 16 listed languages and can detect the browser language automatically.

Pricing

Open-source self-hosted: free. The free plan has no paid plans or usage limits stated. It suits operators prepared to host and configure the project themselves; it does not come with a stated managed-service tier. A Gemini API key is required only for Nova and AI Analysis, not for the rest of the application.

Platforms

Spider-Lens supports Linux, macOS, Windows, web, and self-hosted deployment. It is available as a standalone Node.js application or a native WordPress plugin. The standalone app requires Node.js 18 or later, npm 9 or later, and access to Apache or Nginx log files. The WordPress plugin does not require Node.js on the server, which may make it the more suitable route for WordPress operators who want to avoid that runtime requirement.

Who it's for

Choose Spider-Lens if you need log-based traffic and crawler analysis alongside sitemap inspection, and can manage a self-hosted deployment. It is particularly relevant to operators who want configurable bot and error alerts, per-URL response-time reporting, and exports in one project. Look elsewhere if you need a hosted service, or if you cannot provide access to server logs; the standalone route is also a poor fit where its Node.js and npm requirements cannot be met.

Pros and cons

• Broad scope in a free self-hosted project: log dashboards, crawler monitoring, sitemap crawling, performance analysis, and alerts sit together, without a stated paid tier or usage cap.
• Useful operational outputs: per-URL TTFB, configurable slow thresholds, CSV and Excel exports, and alerts for specific error and crawler conditions support investigation beyond headline traffic counts.
• Deployment choice: the WordPress plugin avoids requiring Node.js on the server, while the standalone app serves operators who can meet its runtime requirements.
• Setup and access are the trade-off: users must self-host and provide Apache or Nginx log access; the standalone app also requires Node.js 18+ and npm 9+.
• AI is optional, not built-in without setup: Nova and AI Analysis need a Gemini API key, though the rest of the application works without it.
• Security needs attention at deployment: the default JWT secret and admin password should be changed before use.

Alternatives

For a broader comparison, see SEO Log File Analysis Tools.

• SEO Log Auditor is another free, open-source option that runs locally; choose it if a locally run log auditor better fits your workflow.
• Screaming Frog SEO Log File Analyser offers a free plan capped at 1,000 log event lines and one project, plus a paid plan at $139 a year; choose it if its project-based licence and stated free cap suit your needs.
• LogLens is a freemium web, API, and self-hosted option; choose it if those deployment formats fit better.
• SEOMER is a freemium web tool with quick page checks and limited daily usage on its free plan; choose it if that narrower workflow is what you need.
• CrawlGenius Log Analyzer has a free web plan capped at 10,000 log lines per month, one project, and a seven-day visible window; choose it if those limits work for your analysis.
• Log File Bot Traffic Cost Analyzer is free and supports Linux, self-hosted, and web use; choose it if that option better matches your platform needs.
• AuditLabs SEO Log File Analyzer is a freemium web tool with a Supporter plan at $7.99 a month; choose it if you prefer that option.
• Crawl Lab is a free web tool with no sign-up, no paid tier, and no server-side size limit; choose it if those terms suit your use.

Verdict

Spider-Lens is a strong choice for technically equipped site operators who want free, self-hosted Apache or Nginx log analysis combined with crawler monitoring, sitemap inspection, and alerts. Its main advantage is that range of tools in one project; its main drawback is the responsibility to deploy it, provide log access, and meet the standalone runtime requirements when using that format. Choose the WordPress plugin to avoid Node.js on the server, or look elsewhere if you need a hosted, managed option.