What GetBotRank's free tools request, and as whom
Our AI Crawler Access Checker and Crawler View tools request pages under the same user-agent strings real AI crawlers use, so we can tell you what those crawlers actually receive. We never do that silently.
How to identify us in your logs
Every request we send appends our own identifier to the vendor's user-agent string, and carries two additional headers:
If a log line has the real bot's product token (e.g. GPTBot) but also carries the (via GetBotRank Free Tools; ...) suffix, that request came from us running a check someone requested against your URL -- not from OpenAI, Anthropic, or any other vendor.
What we do and don't do
- One request per tool run, per user-agent we check -- we don't crawl your site or follow links.
- The Crawler Access Checker deliberately requests your page under each bot UA regardless of what robots.txt says for that token -- that comparison is the point of the tool -- but we always report a robots.txt disallow as the reason a bot is blocked, not the live response.
- Fetches are capped at 5MB, follow at most 5 redirects, and time out after 10 seconds -- we're not a sustained load source.
- We never call an LLM against your content, and we don't store your page's full HTML past the 30-day life of the tool result that referenced it.
How to block us
We don't publish a separate robots.txt token, since we're deliberately requesting pages as the real vendor UAs to report what those crawlers get -- blocking "GetBotRank" specifically wouldn't change what GPTBot or ClaudeBot themselves receive, which is the whole point of the check. If you'd rather we not request your URL at all, block or rate-limit requests carrying the X-GetBotRank-Check header at your CDN or WAF, or email bots@getbotrank.com and we'll add your domain to an exclusion list.
These fetchers only ever run in response to someone submitting a URL on a GetBotRank free tool. We don't run scheduled or background crawls of any site.