Paste a robots.txt file and evaluate whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and other AI crawlers are allowed or blocked on a chosen path | including when they only inherit the * group.
Required fields on the left
Free local parse. Pair with llms.txt Generator for AI crawler snippets.
Appears after you run
See who can crawl
Paste robots.txt to evaluate GPTBot, ClaudeBot, Google-Extended, and other AI agents on a path.
An AI crawler robots.txt checker pastes your robots.txt and reports whether named bots — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and others — are allowed or blocked on a path, including when they only inherit the * group. robots.txt is a convention, not a contract. Bots can ignore it.
Empty Disallow means allow all for that group. The longest matching Allow or Disallow wins for a path. Pair blocks with Terms of Service and an llms.txt policy if you care about the record.
Paste a robots.txt file and evaluate whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and other AI crawlers are allowed or blocked on a chosen path | including when they only inherit the * group.
Paste robots.txt, optionally set a path (default /), then read allow/block per crawler. Fix gaps in the file, then generate llms.txt if you want a preferred-doc hint. This page does not crawl your site.
Paste your robots.txt contents.
Optionally set a path to test (default /).
Review allow/block status per AI crawler.
Fix gaps, then generate snippets with llms.txt Generator.
No. It is a request, not a lock. Some crawlers honor it; some do not. If you need a legal position, use Terms of Service and access controls — not this file alone.
It inherits User-agent: * when that group exists. An explicit User-agent group for that bot overrides the wildcard. If you have no * group and no named group, behavior is “unspecified,” which many crawlers treat as allowed.
An AI crawler robots.txt checker pastes your robots.txt and reports whether named bots — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and others — are allowed or blocked on a path, including when they only inherit the * group. robots.txt is a convention, not a contract. Bots can ignore it.
No. robots.txt is a convention, not a contract. Pair blocks with Terms of Service and an llms.txt policy statement.
It inherits rules from User-agent: * when present. Explicit groups override the wildcard for named agents.
Paste robots.txt and see which AI crawlers are allowed or blocked on a path.
Keep measuring in the same cluster — or jump to the next decision.
robots.txt AI Crawler Checker lives at omnikitapp.net/tools/robots-ai-crawler-checker. Paste a robots.txt file and evaluate whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and other AI crawlers are allowed or blocked on a chosen path | including when they only inherit the * group. Use it as an operator page, then confirm invoices, policies, or published facts on the source. OmniKit does not sell detector evasion or ranking guarantees.
