The robots.txt lines that hide you from AI answers
Blocking training bots is a choice; blocking search bots by accident is the most common fix we find.
The short answer
Many sites block AI search crawlers by mistake while trying to block AI training. OpenAI’s GPTBot collects training data, but OAI-SearchBot fetches pages for ChatGPT search answers; blocking it removes you from those citations. Check robots.txt, your CDN’s bot settings and your server logs for each AI user agent.
What is the difference between AI training bots and search bots?
| Company | Training or general crawl | Search and answers | User-initiated fetch |
|---|---|---|---|
| OpenAI | GPTBot | | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | PerplexityBot | Perplexity-User | |
Google-Extended (a control token for Gemini use) | AI Overviews and AI Mode use the regular Googlebot crawl | ||
Check each vendor’s documentation before you change anything; the names and their scopes change.
Which robots.txt mistakes hide you from AI answers?
- A blanket disallow added in 2023 for every bot with AI in its name, including the search bots.
- A CDN bot shield that challenges all automated traffic. The bot gets a challenge page, not your content.
- Content rendered only in JavaScript. Many AI fetchers do not run scripts, so they see an empty shell.
How do you check your own site?
- Read
/robots.txtand list the rule that applies to each AI user agent. - Check your CDN’s bot management settings for blanket challenges.
- Confirm in server logs, or the Lighthouse crawler view, that each bot gets a 200 response on the pages you care about.
A sensible default
Decide training and search separately. A common setup looks like this:
# Answers: let AI search engines read and cite us
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Training: a deliberate choice, not an accident
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /Our sample brand, Ridgeline Outfitters, had OAI-SearchBot blocked by a 2023 rule. Unblocking it is still the first fix on their list.
Questions people ask
Does blocking Google-Extended remove me from AI Overviews?
Google states that Google-Extended controls the use of content for Gemini models; AI Overviews draw on the regular Googlebot crawl. Check Google’s current documentation before relying on it.
Will allowing AI search bots let them train on my content?
Vendors document their search bots separately from training. Allowing OAI-SearchBot, for example, does not opt you into GPTBot training.
