AI crawlers and robots.txt: which bots to allow and block

Several AI companies publish crawlers that visit your site. Your robots.txt decides which ones can access it. Understanding the difference is key to staying visible without giving up control.

The main user agents

• GPTBot (OpenAI): collects content that may be used to train models.
• OAI-SearchBot (OpenAI): used to surface sites in ChatGPT search results.
• ChatGPT-User (OpenAI): visits pages when a user asks ChatGPT to open them.
• PerplexityBot (Perplexity): indexes pages for Perplexity search.
• ClaudeBot (Anthropic): crawls web content for Anthropic.
• Google-Extended (Google): a control token that lets you opt out of some Gemini training and grounding uses without affecting Search.

Training vs search

Training crawlers collect data to improve models. Search and user-agent crawlers fetch pages to answer a live question. If you want to be cited, you generally want search and user-triggered access allowed.

A sensible default

For most businesses that want visibility, allow search-oriented crawlers. If you have licensing concerns about training, you can disallow training-specific agents. Policies change, so check each provider's current documentation.

Do not forget your firewall

A CDN or security tool can block bots even if robots.txt allows them. Test by fetching key pages with the relevant user agents.

This is one of the first things we review in an AI search audit. For background, see what is AI search optimization.