Crawlr
Crawlr is a website crawler and technical SEO audit tool for finding technical issues, crawl inefficiencies, and site architecture problems.
What Crawlr is
Crawlr is a website crawler and technical SEO audit tool for finding technical issues, crawl inefficiencies, and site architecture problems. It discovers URLs through internal links, sitemaps, canonicals, and JavaScript, with controls for robots.txt compliance, rate limiting, inclusion and exclusion rules, and crawl depth. Its analysis covers response codes, indexability signals, metadata, headings, word counts, and internal and external links. Users can explore results in interactive graphs, directory trees, 3D site views, and content-cluster visualizations. Crawls can be scheduled daily, weekly, or on custom schedules, with email notifications on completion. The API can start crawls, check status, return structured JSON results, access historical data, and send webhook events. The Free plan is free and includes 500 tokens per month, one concurrent crawl, CSV and JSON exports, and 15-day crawl history. Unauthenticated sandbox crawls are limited to 25 pages.
Good
- Discovers URLs through links, sitemaps, canonicals, and JavaScript.
- Crawl controls include depth and URL rules.
- Provides visual maps of site structure and content clusters.
- Scheduled crawls can send completion emails.
- API returns structured JSON and historical data.
Watch out
- Unauthenticated sandbox crawls are limited to 25 pages.
- Free plan includes 500 tokens per month.
Who it suits
Crawlr suits site owners and SEO teams that need technical crawl analysis, visualizations, or scheduled audits. Its plan descriptions also address agencies, growing teams, and high-volume AI and SEO pipelines.
Verdict
Crawlr combines technical SEO analysis with crawl controls, visualizations, scheduling, and API access. The free plan has a monthly token allowance, while larger listed plans start at $8/mo (annual).
Plans and prices
From Crawlr's own pricing page, checked 15 Oct 2026.
| Plan | Price | What it gives you |
|---|---|---|
| Free | Free | 500 tokens/month · 1 concurrent crawl · 500 pages/crawl · 15-day crawl history · full crawling · SEO intelligence and DIQ score · CSV/JSON exports · monthly |
| Starter | $8 a month | 5,000 tokens/month · 5 concurrent crawls · 5,000 pages/crawl · 3-month crawl history · scheduled crawls · PageSpeed Insights · 8 visualizations · yearly, 2 months free |
| Standard | $73 a month | 35,000 tokens/month · 25 concurrent crawls · 10,000 pages/crawl · 1-year crawl history · LLM-ready markdown · advanced indexability · priority queues · yearly, 2 months free |
| Growth | $291 a month | 100,000 tokens/month · 100 concurrent crawls · 25,000 pages/crawl · 1-year crawl history · webhook notifications beta · bulk exports · dedicated account manager · custom crawl frequency · yearly, 2 months free |
| Enterprise | Priced on request | Unlimited pages · custom crawl logic · dedicated resources · SLA · custom pricing |
The facts
Each one read from Crawlr's own pages, with the date it was checked.
What it does
- AI-oriented output
- Crawlr provides token-aware markdown summaries intended for LLM and RAG ingestion.checked 15 Oct 2026
- API
- The API supports starting crawls, checking status, retrieving structured JSON results, accessing historical data, and webhook events.checked 28 Sep 2026
- API access
- Yeschecked 20 Sep 2026
- API capabilities
- The API can start crawls, poll status, retrieve JSON results, access historical crawls and emit webhook events.checked 15 Oct 2026
- API design
- The maker describes a REST API using API-key authentication, standard HTTP methods, documented JSON structures, rate limits and descriptive errors.checked 15 Oct 2026
- Crawl controls
- Crawl settings include robots.txt compliance, rate limiting, include and exclude rules, and crawl depth controls.checked 28 Sep 2026
- Crawl graphs
- Interactive 2D force graphs, 3D graphs, tree views and filtered views visualize URL relationships and site structure.checked 15 Oct 2026
- Data collected
- The service collects account details, billing information, usage logs, device information and crawl data such as URLs, content, metadata, links, status codes and performance metrics.checked 15 Oct 2026
- Data retention
- Personal data is retained while an account is active or needed for service delivery, while crawl-data retention follows the subscription plan.checked 15 Oct 2026
- Deployment
- The official pages describe Crawlr as cloud-based and provide no self-hosting option.checked 15 Oct 2026
- DIQ score
- Its Domain Intelligence Quotient combines indexability, technical health, content quality, link structure and mobile or speed signals into a score from 0 to 100.checked 15 Oct 2026
- Documentation depth
- The API page provides endpoint-level capabilities, data-format descriptions, use cases and authentication, rate-limit and error-handling notes, but no separate public SDK is stated.checked 15 Oct 2026
- Exports
- The Free plan includes CSV and JSON exports; paid plans add LLM-ready markdown exports and higher-tier bulk export capabilities.checked 28 Sep 2026
- Full-site crawling
- It crawls entire websites, including subdomains, parameterized URLs and XML-sitemap URLs.checked 15 Oct 2026
- HTTP analysis
- The crawler tracks 2xx, 3xx, 4xx and 5xx responses and reports redirect chains, loops, soft 404s and broken links.checked 15 Oct 2026
- Independent listing status
- G2 says Crawlr has not yet been reviewed and lacks enough reviews to provide buying insight.checked 15 Oct 2026
- Independent pricing availability
- G2 states that pricing details are unavailable on its listing and directs buyers to the vendor site.checked 15 Oct 2026
- Independent product description
- G2 describes Crawlr as a free web-scraping tool for extracting and managing website data, which differs from crawlr.ai's technical-SEO positioning.checked 15 Oct 2026
- Independent seller identity
- G2 attributes the listed Crawlr product to Crawlr Labs.checked 15 Oct 2026
- Indexability analysis
- It checks noindex, nofollow and canonical directives and flags conflicts such as noindex combined with another canonical target.checked 15 Oct 2026
- Integration
- Google PageSpeed Insights is listed as the named third-party integration for performance analysis.checked 15 Oct 2026
- Integrations
- The pricing page lists Google PageSpeed Insights for the Starter plan and webhook notifications (Beta) for Growth.checked 28 Sep 2026
The company
- Company listing
- A LinkedIn page named Crawlr Labs describes a privately held technology company with 11-50 employees focused on competitor monitoring; the page does not identify crawlr.ai's SnappyTech maker.checked 15 Oct 2026
Money
- Free plan
- Yeschecked 20 Sep 2026
Alternatives to Crawlr
- 1
DefiniteSEO
Free planOpen sourceAndroidiOSWebAPI9.0From $4.99 a month - 2
Marketing Miner
Free planAndroidiOSWebAPI8.8From €19 a month - 3
Rank Math SEO
Free planOpen sourceWebAPISelf-hosted8.7Free plan - 4
Yoast SEO
Free planOpen sourceWebAPISelf-hosted8.7From $118.80 a year - 5
Screaming Frog SEO Spider
Free planAPILinuxmacOSSelf-hosted8.6Free plan - 6
SEO PowerSuite
Free planAPILinuxmacOSWindows8.5From $349 a year