crawler.sh
The facts below are what the maker publishes about crawler.sh, checked and dated. Our editors have not written about it yet.
Plans and prices
From crawler.sh's own pricing page, checked 12 Oct 2026.
| Plan | Price | What it gives you |
|---|---|---|
| Free | Free | Up to 50 pages per session · SEO analysis with CSV/TXT export · JSON and Sitemap XML export · No account required |
| CLI Pro | $99 a year | Up to 10,000 pages per session · NDJSON, JSON, and Sitemap XML output · Markdown content extraction · 16-category SEO analysis with CSV/TXT export · $99 / year |
| Desktop Pro | $99 a year | Visual dashboard with 8 interactive cards · Real-time crawl feed · SEO issues panel with per-URL details · Markdown content archive export · $99 / year |
The facts
Each one read from crawler.sh's own pages, with the date it was checked.
What it does
- AI content
- It extracts article content as clean Markdown with page metadata such as word count, author byline, language, and excerpt for RAG, fine-tuning, or agent context.checked 12 Oct 2026
- Cloud API
- Crawler Cloud is described as a hosted crawling API with scheduled crawls, webhooks, and a dashboard, marked as coming soon.checked 12 Oct 2026
- Crawl behavior
- The crawler respects robots.txt and adapts per-host pacing after 429 and 403 responses.checked 12 Oct 2026
- Data handling
- The maker says crawl data, results, and exported files stay on the user's computer and that it does not collect crawl targets, results, or extracted website content.checked 12 Oct 2026
- Integrations
- The documentation describes an MCP server for connecting crawler.sh output to tools and lists Claude Code, Claude Desktop, Cursor, OpenCode, and Zed as clients.checked 12 Oct 2026
- Interfaces
- The product includes a CLI for automation and scripting and a desktop app with a visual dashboard; both use the same crawling engine.checked 12 Oct 2026
- JavaScript rendering
- A built-in JavaScript engine renders React, Vue, Next, and similar single-page apps without headless Chrome.checked 12 Oct 2026
- Product
- crawler.sh is a local web crawler that extracts website content, analyzes SEO issues, and exports results in standard formats.checked 12 Oct 2026
- Security
- The privacy policy says device-to-API communication uses HTTPS/TLS and authentication tokens are generated with secure random values and expire after a set period.checked 12 Oct 2026
- SEO checks
- The SEO analysis runs 24 automated checks on successful HTML pages and can export findings as CSV or TXT.checked 12 Oct 2026
- Site audit
- Yeschecked 21 Sep 2026
- Support
- Support is offered by email, with typical replies within one business day; the maker says it does not offer live chat, phone support, ticket SLAs, or dedicated account managers.checked 12 Oct 2026
Money
- Free plan
- Yeschecked 21 Sep 2026
Where it runs
- Windows support
- A native Windows desktop build is in progress, and the maker says its Linux CLI binary can run on Windows through WSL2.checked 12 Oct 2026
Alternatives to crawler.sh
- 1
DefiniteSEO
Free planOpen sourceAndroidiOSWebAPI9.0From $4.99 a month - 2
Marketing Miner
Free planAndroidiOSWebAPI8.8From €19 a month - 3
Rank Math SEO
Free planOpen sourceWebAPISelf-hosted8.7Free plan - 4
Yoast SEO
Free planOpen sourceWebAPISelf-hosted8.7From $118.80 a year - 5
Screaming Frog SEO Spider
Free planAPILinuxmacOSSelf-hosted8.6Free plan - 6
SEO PowerSuite
Free planAPILinuxmacOSWindows8.5From $349 a year