What is an AI crawler?
AI crawlers are programs that fetch web pages automatically to collect content for AI systems. They serve three purposes: training language models, building the search index of AI search systems, and fetching individual pages live during a user request. Well-known AI crawlers are GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, and PerplexityBot.
Which AI crawlers exist?
Six identifiers appear most often in log files and robots.txt discussions:
| Identifier | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Collects content for training the GPT models |
| OAI-SearchBot | OpenAI | Builds the index of ChatGPT search |
| ChatGPT-User | OpenAI | Fetches pages live when a user asks for them in ChatGPT |
| ClaudeBot | Anthropic | Collects content for training the Claude models |
| PerplexityBot | Perplexity AI | Builds the search index of Perplexity |
| Google-Extended | Not a crawler of its own but a control token: governs whether your content is used for training the Gemini models |
Unlike a classic crawler such as Googlebot, many of these bots do not work for a search index but for a language model. That is exactly why it pays to check the purpose of each identifier before blocking it.
How do you control AI crawlers via robots.txt?
You control AI crawlers like any bot through the robots.txt in the root directory of your domain. Every rule applies exactly to the named identifier:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /This example forbids OpenAI from collecting content for model training and at the same time allows inclusion in ChatGPT search. Important for your expectations: robots.txt is an instruction, not a barrier. Reputable providers comply, rogue scrapers do not.
Why does blocking GPTBot not remove you from ChatGPT search?
Because GPTBot and OAI-SearchBot are independent switches. OpenAI documents the separation explicitly: each bot only controls its own purpose, and a site owner can allow OAI-SearchBot while disallowing GPTBot. In practice this means:
- GPTBot blocked, OAI-SearchBot allowed: Your content does not flow into model training, but your website still appears in the answers of ChatGPT search.
- OAI-SearchBot blocked: According to OpenAI, your website is no longer shown in ChatGPT search answers, regardless of what applies to GPTBot.
- ChatGPT-User: Fetches pages on direct user request. Because the action is initiated by a human, robots.txt rules may not apply here according to OpenAI.
The common shortcut “block GPTBot = out of ChatGPT” is therefore wrong in both directions: the training block costs no visibility in AI search, and whoever wants to disappear has to hit the right switch.
Should you block or allow AI crawlers?
For most companies the answer is: Allow them, at least the search crawlers. Answers from ChatGPT, Perplexity, and AI search are a growing visibility channel; blocking OAI-SearchBot or PerplexityBot means giving up mentions and source links. A training block via GPTBot or Google-Extended is a separate decision, for example for exclusive content.
If you allow AI crawlers, you can make their work easier: an llms.txt summarizes the most important pages of a domain as a briefing, and citably structured content raises the chance of being mentioned. Both belong to Generative Engine Optimization (GEO) and are the core of our optimization for AI visibility.
Frequently asked questions about AI crawlers
Is Googlebot an AI crawler?
No. Googlebot is Google’s classic search crawler; its data, however, also feeds AI Overviews and AI Mode. For training the Gemini models, Google offers a separate robots.txt switch called Google-Extended, which does not affect Google Search.
Do AI crawlers respect robots.txt?
Major providers such as OpenAI, Anthropic, and Google document their crawlers and state that they respect robots.txt. The file is not a technical barrier, though: rogue bots ignore it. If you want to protect content strictly, you need server-side blocks.
How do I recognize AI crawlers in my log files?
By the user agent: identifiers such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, or PerplexityBot appear in the user agent string of every request. The providers also publish IP lists that let you distinguish real crawlers from impostors.
Welt der SEO lernen?
