Crawlie

Overview

Crawlie is a technical crawler for SEO, generative-engine optimization, and accessibility. It will suit developers and SEO teams who want audits to feed into reports or automated workflows. The strongest case for it is the combination of distinct scores, export formats, and agent interfaces; hosted users should weigh that against monthly credit and crawl limits.

Key features

Crawlie checks broken links, redirects, missing metadata, and more than 40 SEO and generative-engine issues. JavaScript rendering and structured data audits extend its reach beyond a basic link check. Each crawl returns separate Health, GEO, and Accessibility scores, helping teams distinguish areas to investigate rather than relying on one overall grade.

Results can be output in the terminal or as JSON, CSV, and shareable HTML. That gives technical users machine-readable options alongside a format suited to sharing reports. The open-source crawler includes an MCP server; the hosted service adds a hosted MCP server and REST API on every plan, useful for agent workflows and integrations.

Hosted deploy-triggered crawls are documented for Vercel, with GitHub, Netlify, and custom CI deploy hooks also described. The Vercel integration requests permission to read the project list and says it does not read code, deployments, or environment variables, or write to the Vercel account. The CLI and MCP server install through npm, and the desktop app has a signed macOS DMG.

Pricing

The Free plan costs $0 per month and includes 200 credits per month, three sites, up to 1,000 pages per crawl, and one seat. Community support is the trade-off; the modest credit and site allowances make this a sensible way to evaluate hosted use, but not a large-crawl allowance.

Pro is $29 per month, or $24 per month billed annually. It raises the quota to 2,000 credits per month and 250,000 pages per crawl, with unlimited sites and seats. Email support is included. This fits teams that have outgrown the Free caps and need room for larger crawls without restricting collaborators.

Business is $149 per month, or $124 per month billed annually. It provides 10,000 credits per month, up to 250,000 pages per crawl, unlimited sites and seats, and priority support. The higher credit allowance is the main step up from Pro; it is better suited to heavier recurring use than to a small team with occasional audits.

Scale has custom pricing, custom credit and page caps, custom AI prompt volume and cadence, SAML SSO, and dedicated support with onboarding. It is aimed at organizations that need tailored capacity or those enterprise-oriented terms. Pro and Business also include unlimited sites and seats, so Scale's case rests on customization and support rather than basic team access.

Platforms

Crawlie supports API, Linux, macOS, web, and Windows. The npm-installable CLI and MCP server suit command-line workflows, while the signed DMG provides a desktop option for macOS users.

Who it's for

The documented use cases include pre-launch QA, GEO optimization, agent workflows, CI/CD gating, and client reporting. Crawlie is a good fit when a team wants technical checks alongside GEO and accessibility scores, or needs reports and crawler access to connect with its workflow. It is less suitable for users who need generous hosted capacity at no cost or have no use for its GEO and agent features.

Pros and cons

• Pros: More than 40 SEO and generative-engine checks, JavaScript rendering, and structured data audits cover more than basic link crawling.
• Pros: Separate Health, GEO, and Accessibility scores, plus terminal, JSON, CSV, and HTML reports, support analysis and sharing.
• Pros: An open-source MCP server and hosted MCP server and REST API give both local and hosted workflows an agent interface.
• Cons: Free hosted use is limited to 200 credits per month, three sites, one seat, and 1,000 pages per crawl.
• Cons: Pro and Business have the same 250,000-page crawl ceiling; Business chiefly adds credits and priority support.
• Cons: Scale's custom pricing and caps make its cost and capacity less predictable than the fixed plans.

Alternatives

For a directory-wide comparison, browse Technical SEO Software.

• WebCrawler Pro is another free and freemium crawler option, with Linux, macOS, self-hosted, web, and Windows platforms; its free tiers allow 50 requests per day as a guest or 100 when signed in.
• Crawlix may suit a user seeking a free desktop crawler with a 100-page crawl cap, one project, one device, and two concurrent connections; Pro is $15 a month.
• Screaming Frog SEO Spider is an alternative for users who want a free crawl capped at 500 URLs or a paid licence for unlimited URLs.
• A1 Website Analyzer offers a free mode that continues after a 30-day evaluation period, with Windows and macOS support.
• Netpeak Spider offers a free Windows and macOS plan capped at 500 URLs and five threads.
• Screaming Frog SEO Log File Analyser is the more focused alternative for log analysis, with a free plan capped at 1,000 log event lines and one project.
• Crawlr offers a free web and API plan with 500 tokens per month, one concurrent crawl, full-site crawling, CSV and JSON exports, and 15-day crawl history.
• CrawlX offers a free web and API tier with 500 URLs per crawl, five crawls per month, one project, 150+ SEO checks, and CSV, JSON, and PDF exports.

Verdict

Choose Crawlie if you want technical SEO crawling combined with GEO and accessibility scoring, flexible reports, and MCP or API access for automation. The Free plan provides a useful entry point, but its credits, site count, and crawl size are tight; teams with recurring or larger needs should budget for Pro or Business. If those hosted limits matter more than its distinct scores and agent workflows, compare alternatives before committing.