Skip to content

robots.txt Auditor

Paste any URL. Get a one-page audit of its crawl directives, compliance posture, and red flags.

What this checks

  • Reachability — does the site even publish a robots.txt at the standard location?
  • Crawl directives — user-agent groupings, allow/disallow rules, crawl-delay values
  • Sitemap declarations — does the file point to discoverable sitemaps?
  • Wildcard blocks — paths blocked for all user-agents
  • Compliance flags — patterns that often indicate misconfiguration

Powered by ComplyCrawl

This tool runs against ComplyCrawl's robots.txt evaluator — the same module that gates every fetch in the full product. If you crawl the web for SEO monitoring, catalog watching, or any commercial purpose, you need this check running before every request.

See ComplyCrawl product page →

Note: this audit fetches the target's /robots.txt from your browser. We don't log it. If you need audit-grade, tamper-evident logging of these checks at scale, that's what the full ComplyCrawl codebase provides.