Robots.txt tester for AI crawlers
Paste a page URL and see which of 20+ crawlers your robots.txt allows to fetch it, and exactly which rule decided each answer.
How it works
- 1
We fetch your robots.txt
We request /robots.txt from your site with our own honest user agent and parse every group and rule.
- 2
We apply the real precedence rules
Crawlers pick the group that names them, then the longest matching Allow or Disallow rule. If an Allow and a Disallow match equally, Allow wins. We follow that exactly.
- 3
We test the path against each bot
You see allowed or blocked for every crawler, the rule that decided it, and a fix when search or live-browsing bots are blocked.
Why it matters for AI search
AI search products use separate crawlers for training, search indexes and live browsing. Blocking a training crawler is a business choice. Blocking a search or live-browsing crawler removes you from the answers people are getting right now. A single stray Disallow can do that without anyone noticing.
Frequently asked questions
Which crawlers does it test?
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot and several more.
What happens when two rules both match?
The longest matching rule wins. If an Allow and a Disallow are the same length, Allow wins. Wildcards (*) and the end anchor ($) are supported.
Does this tell me if my CDN blocks bots?
No. This tool reads only robots.txt. A firewall or CDN rule can block a bot even when robots.txt allows it. The AI Crawler Access Checker makes live requests to catch that.
What if I have no robots.txt?
Crawlers then assume everything is allowed. We flag it as a warning so you can add one deliberately, with a Sitemap line.
Is blocking GPTBot bad for AI search visibility?
Not by itself. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot and live browsing uses ChatGPT-User, and those are the ones that affect whether you appear in answers.