Robots.txt Tester: Check 20+ AI and Search Crawlers on Any URL
A free robots.txt tester that checks one URL against GPTBot, ClaudeBot, PerplexityBot, Googlebot and 17 more crawlers, and shows the exact rule that decided each answer.
A robots.txt file is short, and that is the problem. One line like Disallow: /blog can quietly take a whole section out of AI search, and nothing on your site will tell you.
Most testers check Googlebot and stop there. AI search uses a different set of crawlers, and they do different jobs: some collect training data, some build a search index, and some fetch a page live when a person asks a question. Blocking the first kind is a choice. Blocking the other two costs you answers.
How to use it
- Paste the full URL of a page you care about, such as your pricing page or a top blog post.
- Press the button. We fetch your robots.txt and test the path of that URL against more than 20 crawlers.
- Read the table. Each crawler shows allowed or blocked, its purpose, and the exact rule that decided it.
What the checks mean
- Search crawlers. OAI-SearchBot, PerplexityBot, Googlebot and Bingbot build the indexes that answer engines draw from. Blocked here is a fail.
- Live-browsing agents. ChatGPT-User, Claude-User and Perplexity-User fetch a page when someone asks about it. Blocked here means that question gets no answer from you.
- Training crawlers. GPTBot, ClaudeBot and CCBot collect training data. Blocking them is a legitimate decision, so we report it without marking it as a problem.
When two rules match, the longest one wins, and on a tie Allow wins. That is how Google and Bing read the file, and it is how the tester reads it too. Wildcards and the $ end anchor are supported.
Try it
Open the tool at getbotrank.com/tools/robots-txt-tester. It is free, needs no signup, and uses no AI model. If you want to know whether AI engines actually mention your brand, the full AI visibility scan tests that directly. You can also browse all free tools.