Is your site in AI training data?
Common Crawl is the open web archive many language models learn from. Check whether your domain is in the latest crawls.
How it works
- 1
We query the Common Crawl index
We check the three most recent crawls for your domain and its subdomains.
- 2
We show what we found
Which crawls include you, a size signal and sample URLs.
- 3
We check CCBot in robots.txt
CCBot is Common Crawl's crawler. If you block it, future crawls skip you.
Why it matters for AI search
Many large language models were trained at least partly on data derived from Common Crawl. Being absent from it is one reason a model may know little about a brand. Being present is not a guarantee, and blocking CCBot is a legitimate choice.
Frequently asked questions
Does being in Common Crawl mean ChatGPT knows my site?
It raises the odds that data from your site was available during training. It does not prove any particular model used it.
Why might I be missing?
New sites, sites blocking CCBot, pages with few inbound links and slow servers are the common reasons.
Should I allow CCBot?
If you want to be in open datasets, yes. If you do not want your content used for training, blocking is a reasonable choice and past crawls remain in the archive.
What does the index page count mean?
It is the number of index blocks that hold your domain, a rough size signal rather than an exact page count.