OmniKit
← Tools

Digital Marketing

Free tool

robots.txt AI Crawler Checker

Paste a robots.txt file and evaluate whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and other AI crawlers are allowed or blocked on a chosen path — including when they only inherit the * group.

  1. 1Fill inputs
  2. 2Run
  3. 3Copy / export

Inputs

Free local parse. Pair with llms.txt Generator for AI crawler snippets.

Results

See who can crawl

Paste robots.txt to evaluate GPTBot, ClaudeBot, Google-Extended, and other AI agents on a path.

How it works

From inputs to a decision

Paste a robots.txt file and evaluate whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and other AI crawlers are allowed or blocked on a chosen path — including when they only inherit the * group.

  1. 01

    Paste your robots.txt contents.

  2. 02

    Optionally set a path to test (default /).

  3. 03

    Review allow/block status per AI crawler.

  4. 04

    Fix gaps, then generate snippets with llms.txt Generator.

Tips for better results

  • Empty Disallow means allow all for that group.
  • Longest matching Allow/Disallow wins for a path.
  • Treat AI answers as snapshots — re-run after content or schema changes.

Frequently asked questions

Do bots always obey robots.txt?
No. robots.txt is a convention, not a contract. Pair blocks with Terms of Service and an llms.txt policy statement.
What if a bot is not listed?
It inherits rules from User-agent: * when present. Explicit groups override the wildcard for named agents.

Related tools

Keep measuring in the same cluster — or jump to the next decision.