AI Crawlers Are Already Here: A Practical Guide to robots.txt and llms.txt

Before a model can cite your page, something has to fetch it. Bot traffic from AI crawlers and agents has grown fast enough that several networks now report it as a substantial share of all requests they see - and a meaningful chunk of that is crawlers building or refreshing the retrieval indexes that power AI Overviews, ChatGPT browsing, Perplexity, and Claude. If those bots can't reach your pages, none of the structural work you do to make content citable ever gets read in the first place.

This is the part of AEO/GEO that has nothing to do with writing and everything to do with configuration - and it's where we see the most avoidable mistakes.

The crawlers actually worth knowing by name

Each major AI product runs its own crawler, identified by its own user agent string:

• GPTBot - OpenAI's crawler, used to train and ground models including ChatGPT's browsing.
• ChatGPT-User - a separate OpenAI agent that fetches pages live, in response to a specific user query.
• ClaudeBot / anthropic-ai - Anthropic's crawlers for Claude.
• PerplexityBot - Perplexity's crawler for its answer engine.
• Google-Extended - controls whether Google can use your content for Gemini and AI Overviews, separately from classic Googlebot indexing.

These are distinct from the classic search crawlers (Googlebot, Bingbot) you've likely already allowed. Blocking classic search crawlers has always been an obvious mistake. Blocking these newer ones is easy to do by accident, because most robots.txt files predate their existence.

Where sites accidentally block themselves

We see the same handful of misconfigurations repeatedly during audits:

1. Inherited or copy-pasted robots.txt rules. A blanket Disallow: / left over from a staging environment, or a boilerplate file copied from another project, can silently block every AI crawler along with everything else.
2. CDN or WAF-level bot blocking. Cloudflare, and similar edge providers, ship dedicated toggles for blocking "AI bots" or "AI scrapers" as a category - often defaulted on for sites that opted into bot-fighting features aimed at scrapers, not at answer engines. This blocks the crawler at the network edge, so it won't even show up as a robots.txt issue when you check the file itself.
3. Rate limiting that treats crawlers as abuse. Aggressive rate limits built to stop scraping bots can throttle or ban legitimate AI crawlers making repeated, spaced-out requests, especially on sites with strict per-IP thresholds.
4. Confusing "opt out of AI training" with "opt out of citation." Some teams deliberately block GPTBot to keep content out of model training data, without realizing that separate agents like ChatGPT-User still need access to fetch and cite a page live. Training opt-out and citation eligibility are different settings, not one switch.

The fix for all four is the same first step: check what's actually happening, not what you assume is configured. Fetch your robots.txt directly, check your CDN's bot management settings for an AI-bot category, and pull recent server logs filtered by these user agent strings to confirm the crawlers are actually reaching your pages and getting a 200, not a 403 or a redirect loop.

Where llms.txt actually fits

llms.txt - a proposed markdown file at your site root that summarizes your content for AI systems, similar in spirit to a sitemap - has gotten a lot of attention as "the new robots.txt." The honest state of it right now: no major AI lab has confirmed that their crawlers read or use it, and Google's own guidance explicitly lists llms.txt among the tactics it doesn't consider necessary for AI visibility.

That doesn't mean skip it. It's cheap to create, costs nothing to maintain alongside a sitemap, and if adoption grows, you'll already have it. Just don't treat it as a substitute for the things that are confirmed to matter: crawlability, clean semantic structure, and a robots.txt that isn't accidentally blocking the bots you want reading your pages. Spend the real effort there first.

What to check this week

1. Fetch yourdomain.com/robots.txt and confirm GPTBot, ChatGPT-User, ClaudeBot, and Google-Extended aren't disallowed.
2. Check your CDN/WAF dashboard for an "AI bots" or "AI scrapers" blocking category and confirm it's off, or scoped to the bots you actually intend to block.
3. Pull a week of server logs and grep for these user agents - confirm they're getting 200s, not 403s, 429s, or redirect loops.
4. Add an llms.txt if you have the bandwidth, but treat it as a nice-to-have, not a fix for a crawlability problem.

None of this guarantees a citation. It's the qualifying round - the same role technical SEO fundamentals play in the broader AEO/GEO pipeline. A Page Monitor that re-scans your key pages weekly will catch a new block before it costs you weeks of invisible crawl gaps - which is often a faster way to notice a bad WAF rule than staring at robots.txt by hand.