Four assistants, one foundation What each one actually reads, and what I do about it.
Each company runs its own crawlers with its own rules. Here is what they publish, quoted from their own documentation, and the setting I use on your site.
Claude-SearchBot · Claude-User · ClaudeBot
Anthropic runs three crawlers with three jobs. Claude-SearchBot indexes pages for Claude’s search results, Claude-User fetches a page when someone asks Claude a question that needs it, and ClaudeBot collects content that may be used for model training. Each obeys its own robots.txt rule.
How I build for it. Search and user fetches stay open, so Claude can find and read the page when a customer asks. Training is your call, and we set it the way you want.
OAI-SearchBot · ChatGPT-User · GPTBot
OpenAI says sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers.” GPTBot is the separate training crawler. ChatGPT-User is a visit a person triggers, and OpenAI notes robots.txt rules may not apply to it.
How I build for it. OAI-SearchBot is allowed, because it is the one that decides whether you can be named in ChatGPT’s answers. Blocking GPTBot does not remove you from search, so the two are set separately.
Googlebot · Google-Extended
Google’s documentation says the requirement for AI Overviews and AI Mode is to be indexed and eligible to appear in Search with a snippet, and that there are “no additional requirements.” It also says you don’t need new machine-readable files, AI text files or markup. Google-Extended is the control for AI training and grounding in some of Google’s other systems.
How I build for it. This is plain, hard SEO: a fast page that Google indexes, a complete Google Business Profile, clear content and real reviews. Nothing exotic is needed, so nothing exotic gets sold.
No official crawler documentation
I looked for xAI’s own documentation on a Grok crawler and robots.txt behavior and did not find any. Third-party bot directories list several names and disagree, and one independent test reports Grok traffic using ordinary browser identifiers. I won’t tell you a robots.txt line controls it.
How I build for it. The same fundamentals as above: a fast, crawlable, clearly written site and a presence people actually link to and talk about. When xAI publishes how its crawler works, this row gets updated.
