The short version
- There are two distinct categories: training crawlers and live retrieval crawlers. The decision is different for each.
- Blocking a retrieval bot removes you from answers users are asking right now. Blocking a training bot affects a future model.
Google-Extendedcontrols AI use of your content without affecting Search rankings. It is a separate token.robots.txtcompliance is voluntary. It works for named, well-behaved crawlers — not as protection.- For most marketing sites the sensible default is: allow retrieval, decide on training deliberately.
Two kinds of crawler, two different decisions
This distinction is the whole article, and almost every “should I block AI bots” discussion ignores it.
Training crawlers collect content to train future models. The content goes into a dataset. There is no attribution, no link, no referral traffic, and no benefit to you in the short term. The output is a model that may be better at your subject, released months later.
Live retrieval crawlers fetch pages right now, because a user asked something and the assistant is going to look it up. These typically cite sources, sometimes link, and can send real traffic. If you block them, you are absent from the answer.
Same file, same syntax, opposite consequences. Blocking a training bot is a decision about future models. Blocking a retrieval bot is a decision to be invisible in a surface people are actively using today.
If you take one thing from this: do not block the retrieval crawlers because you read an article about protecting your content from training.
The user-agents that matter
Tokens change and new ones appear, so verify against each provider’s current documentation before relying on this. As of writing:
| User-agent | Operator | Purpose | My default |
|---|---|---|---|
GPTBot | OpenAI | Training data collection | Your call |
OAI-SearchBot | OpenAI | Search index for ChatGPT search | Allow |
ChatGPT-User | OpenAI | Live fetch when a user asks | Allow |
ClaudeBot | Anthropic | Training data collection | Your call |
Claude-User | Anthropic | Live fetch on user request | Allow |
Claude-SearchBot | Anthropic | Search indexing | Allow |
Google-Extended | Gemini training & AI grounding | See below | |
PerplexityBot | Perplexity | Search indexing | Allow |
Perplexity-User | Perplexity | Live fetch on user request | Allow |
Applebot-Extended | Apple | Apple Intelligence training | Your call |
meta-externalagent | Meta | Training data collection | Your call |
CCBot | Common Crawl | Open dataset used by many trainers | Your call |
Bytespider | ByteDance | Training; historically aggressive | Often blocked |
Note the pattern: the major providers now ship separate tokens for training and retrieval precisely so you can make different decisions. Blocking GPTBot while allowing OAI-SearchBot and ChatGPT-User is a coherent, supported position — opt out of training, stay present in answers.
Bytespider is worth calling out separately. The common complaint is not the training use but crawl behaviour — high request volume with inconsistent respect for directives. If your server logs show it hammering you, blocking it is a performance decision as much as a licensing one.
Free tool · Generate it correctly
Write a robots.txt that does what you meant
The generator builds valid directives and flags the classic errors — a named group that silently overrides your global rules, a blocked path with a canonical pointing at it, a sitemap line in the wrong place.
Google-Extended is a special case
This one causes the most confusion, so precisely:
Google-Extended is not a crawler. No bot identifies as it. It is a control token: a way to tell Google that content it has already crawled with Googlebot should not be used for Gemini training or for grounding AI answers.
What follows from that:
- Blocking
Google-Extendeddoes not affect your Search rankings. Googlebot still crawls and indexes you normally. - It may reduce your presence in AI Overviews and Gemini answers. That is the trade-off you are accepting.
- Blocking
Googlebotto avoid AI use is a catastrophic error. It removes you from Google Search entirely. These are separate tokens for a reason.
So the decision is narrow: do you want your content used to ground AI answers on Google surfaces? If yes, allow it. If you would rather not, block Google-Extended specifically and leave everything else alone.
Writing the directives correctly
Same syntax as any other robots.txt rule. Each user-agent gets its own group:
# Allow live retrieval — these fetch because a user asked
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
# Opt out of training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Three syntax traps:
- The most specific matching group wins — a named group completely replaces
User-agent: *for that crawler. It does not inherit. If your wildcard group has tenDisallowlines and yourGPTBotgroup has oneAllow: /, GPTBot can crawl everything, including the ten paths you thought were blocked. - Repeat your global rules inside named groups if you want them to apply there. Verbose, but this is exactly where people accidentally expose admin paths.
- Tokens are case-insensitive but spelling is not.
GPTbot,Claude-BotandGooglebot-Extendedall silently match nothing.
After changing anything, fetch /robots.txt in a browser and read it. Then check your server logs for the user-agents you targeted, and confirm the behaviour changed. A directive you never verified is a directive you should not rely on.
The actual trade-off
Stripped of ideology, here is what you are choosing between.
| Allowing | Blocking | |
|---|---|---|
| Training crawlers | Your expertise may inform future models. No attribution, no traffic, no control. | Your content is excluded from that dataset. No effect on current visibility. |
| Retrieval crawlers | You can be cited and linked in answers people are asking now. Some referral traffic. | You are absent from those answers. A competitor is cited instead. |
The honest position on training is that nobody knows what it is worth. There is no measurable return, and there is a reasonable argument that being well-represented in models that increasingly mediate discovery is worth something. There is an equally reasonable argument that giving away a content library for nothing is a bad deal. It depends on what your content is: a publisher whose archive is the business has a very different calculation from a consultant whose blog is a lead-generation asset.
The position on retrieval is much clearer. Being cited in an answer is closer to ranking than to being scraped. Blocking it is choosing not to appear.
What robots.txt cannot do
Setting expectations, because this is oversold in both directions:
- Compliance is voluntary. Well-behaved crawlers respect it. Anything determined to ignore it will.
- It is not access control. A blocked URL is still publicly reachable. Use authentication for anything that must not be read.
- It cannot claw back what was already collected. Blocking today does not remove your content from a dataset gathered last year.
- It does not stop content reaching models indirectly. If your article is quoted, syndicated or discussed elsewhere, that copy is outside your control.
- Blocking crawling is not the same as blocking indexing. A blocked page can still be listed from external links — and because it is never crawled, a
noindexon it is never read.
What I would do
A defensible default for a marketing site
- Allow all retrieval crawlers. These are answer surfaces you want to appear in. There is no upside to hiding.
- Allow training crawlers too — if your content exists to generate leads rather than to be sold. Broad representation is roughly aligned with your interest.
- Block training selectively if your content is the product: courses, paid research, a licensed archive.
- Block aggressive crawlers on behaviour, based on your own logs, not on someone else’s blocklist.
- Never block
Googlebotorbingbotin an attempt to control AI use. Use the-Extendedtokens. - Re-read it twice a year. Tokens change; stale robots files quietly do the wrong thing.
If you sell services and your blog is there to demonstrate competence, the answer is usually to allow nearly everything and concentrate on being worth citing. The sites that win in AI answers are not the ones with the cleverest robots file — they are the ones whose content states things clearly enough to be quoted.
FAQ
Should I block AI crawlers from my website?
It depends which ones. Blocking live-retrieval crawlers like ChatGPT-User, Claude-User or Perplexity-User removes you from answers users are asking right now, which is rarely in your interest. Blocking training crawlers like GPTBot or ClaudeBot is a genuine judgement call that depends on whether your content is a lead-generation asset or the product itself.
Does blocking Google-Extended hurt my search rankings?
No. Google-Extended is not a crawler and does not affect Googlebot's crawling or indexing at all. It only controls whether your content is used for Gemini training and for grounding AI answers. What you must never do is block Googlebot to try to control AI use — that removes you from Google Search entirely.
Can AI companies ignore my robots.txt?
Technically yes — compliance is voluntary and always has been. The major providers publish their tokens and state that they respect directives, and there is reasonable evidence they do, because ignoring robots.txt is a reputational and legal liability at their scale. But robots.txt is a request, not a barrier. Anything that must not be read needs authentication.
What is the difference between GPTBot and ChatGPT-User?
GPTBot crawls to collect training data for future models — no attribution, no traffic, no immediate benefit to you. ChatGPT-User fetches a page live because a user asked something and ChatGPT is looking it up, which can produce a citation and a click. They are separate tokens specifically so you can allow one and block the other.
Will blocking AI crawlers protect my content from being copied?
No, and it is the wrong tool for that. robots.txt only affects crawlers that identify themselves and choose to comply. It does nothing about scrapers, does not remove content already collected, and cannot stop your material reaching models indirectly through quotes, syndication or discussion elsewhere. It is a licensing signal, not protection.
Thejas Acharya
Digital strategy consultant behind Digital with Thejas, based in Mangaluru, working with founders and marketing teams across India and the US. I build websites, landing pages, Meta Ads campaigns and AI automation that are judged on one thing: whether they generate leads. The free tools on this site are the checks I run on client work, opened up.