Add a group “User-agent: GPTBot” with “Disallow: /” to robots.txt. OpenAI stops using the site to train models; ChatGPT search is not affected — OAI-SearchBot handles it and can stay open. If you also want to stop training by other companies, the generator above puts every training bot into one group.

how to block claudebot

A group “User-agent: ClaudeBot” with “Disallow: /”. According to Anthropic's documentation the bot follows robots.txt, so the file is enough. Claude search has a separate Claude-SearchBot and user requests have Claude-User: blocking ClaudeBot does not touch them. The old names anthropic-ai and Claude-Web are added with the “safety net” checkbox.

does google-extended block ai overviews

No. Google-Extended concerns Gemini model training and grounding in Gemini and Vertex AI and affects neither Google Search nor AI Overviews: the overviews are built on the ordinary Googlebot index. You can limit how your text appears in them with the nosnippet and max-snippet meta tags or the data-nosnippet attribute — the generator suggests ready tags under the file.

why does an ai bot visit my site although it is blocked

Either it is a user-request bot that, according to its documentation, may not follow robots.txt, or a robot posing as a browser. In 2025 Cloudflare reported that Perplexity did this. Only server or CDN rules protect against such visits: checking the user agent, IP ranges from the lists companies publish, and rate limiting.

how often is the ai bot list updated

We check the list against company documentation once a month and show the date of the last check for every entry. New bots appear often, so there is also an extended list from the ai.robots.txt community project: one checkbox adds it to the file. It contains no search or link-preview robots, so the site does not drop out of search.

How to Block AI Bots in robots.txt: 2026 List

01

Generator

Basic rules

Block sections · one path per lineSitemap · one address per line

Clean-param for Yandex · parameters · path

Presets

User-request bots often ignore robots.txt: only server rules close the site to them reliably.

Model training

GPTBot, ClaudeBot, Google-Extended, CCBot…

Blocking affects neither search nor answers with links.

GPTBot

OpenAI

follows robots.txtdocumentation

ClaudeBot

Anthropic

follows robots.txtdocumentation

Google-Extended · token

Google

follows robots.txtdocumentation

Applebot-Extended · token

Apple

follows robots.txtdocumentation

CCBot

Common Crawl

follows robots.txtdocumentation

Meta-ExternalAgent

AI search and citation

OAI-SearchBot, Claude-SearchBot, PerplexityBot, YandexAdditional…

Blocking removes the site from answers with links — that is lost traffic.

User requests

ChatGPT-User, Claude-User, Perplexity-User…

Many of them may ignore robots.txt; only the server is reliable.

Safety net: add the old names anthropic-ai, Claude-Web, FacebookBot — current documentation no longer lists them, but old records still use them131 more bots from the community — not split into classes, and some of them are search bots. List ai.robots.txt · MIT

robots.txt · 46 lines

User-agent: *
Disallow: /admin/
Disallow: /search
# AI: model training — blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: cohere-training-data-crawler
User-agent: Timpibot
Disallow: /
# AI: search and citation — blocked
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
Disallow: /
# AI: user requests — blocked
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
User-agent: Amzn-User
User-agent: MistralAI-User
User-agent: Google-Agent
Disallow: /
# Yandex: Neuro and Alice answers — blocked
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
Disallow: /
Clean-param: utm_source&utm_medium&utm_campaign /
Sitemap: https://example.ru/sitemap.xml

What robots.txt cannot do

Google AI Overviews and Bing chat run on the ordinary search robots — limit them with meta tags in <head>.

Google AI Overviews and AI Mode

<meta name="robots" content="nosnippet"> or max-snippet:0

Bing chat answers

<meta name="robots" content="noarchive, nocache">

02

AI bot list

from official company documentation

Company	User agent	Class	Follows	Checked	Documentation
OpenAI	GPTBot	Training	follows robots.txt	10.10.2026	Open
Anthropic	ClaudeBot	Training	follows robots.txt	10.10.2026	Open
Google	Google-Extended a token, not a separate robot	Training	follows robots.txt	10.10.2026	Open
Apple	Applebot-Extended a token, not a separate robot	Training	follows robots.txt	10.10.2026	Open
Common Crawl	CCBot	Training	follows robots.txt	10.10.2026	Open
Meta	Meta-ExternalAgent	Training	follows robots.txt	10.10.2026	Open
Amazon	Amazonbot	Training	follows robots.txt	10.10.2026	Open
Mistral	MistralAI-Training	Training	follows robots.txt	10.10.2026	Open
ByteDance	Bytespider	Training	no documentation	10.10.2026	none
Cohere	cohere-training-data-crawler	Training	no documentation	10.10.2026	none
Timpi	Timpibot	Training	no documentation	10.10.2026	none
OpenAI	OAI-SearchBot	AI search	follows robots.txt	10.10.2026	Open
Anthropic	Claude-SearchBot	AI search	follows robots.txt	10.10.2026	Open
Perplexity	PerplexityBot	AI search	follows robots.txt	10.10.2026	Open
DuckDuckGo	DuckAssistBot	AI search	follows robots.txt	10.10.2026	Open
Mistral	MistralAI-Index	AI search	no documentation	10.10.2026	Open
Amazon	Amzn-SearchBot	AI search	follows robots.txt	10.10.2026	Open
Meta	meta-webindexer	AI search	follows robots.txt	10.10.2026	Open
Yandex	YandexAdditional by name only, ignores “*”	AI search	follows robots.txt	10.10.2026	Open
Yandex	YandexAdditionalBot by name only, ignores “*”	AI search	follows robots.txt	10.10.2026	Open
OpenAI	ChatGPT-User	User requests	may not follow	10.10.2026	Open
Anthropic	Claude-User	User requests	follows robots.txt	10.10.2026	Open
Perplexity	Perplexity-User	User requests	may not follow	10.10.2026	Open
Meta	Meta-ExternalFetcher	User requests	may not follow	10.10.2026	Open
Amazon	Amzn-User	User requests	may not follow	10.10.2026	Open
Mistral	MistralAI-User	User requests	no documentation	10.10.2026	Open
Google	Google-Agent	User requests	may not follow	10.10.2026	Open

03

About the list

Every AI company names its robots in its own way and publishes crawling rules. We put them in one table and noted for each what it does and how it treats robots.txt — according to official documentation. Where there is no documentation, the table says so; every entry carries the date it was checked.

OpenAI

Three bots

GPTBot

GPTBot trains models, OAI-SearchBot brings the site into ChatGPT search, ChatGPT-User opens pages at a user's request and, according to OpenAI's documentation, may not follow robots.txt.

Anthropic

Three bots

ClaudeBot

ClaudeBot trains models, Claude-SearchBot handles search, Claude-User handles user requests. According to the docs, all three follow robots.txt.

Google

A token

Google-Extended

There is no separate AI robot: Google-Extended is a flag for training and grounding in Gemini and Vertex AI. AI Overviews are controlled by the nosnippet and max-snippet meta tags.

Perplexity

Search

PerplexityBot

The company's search bot. In 2025 Cloudflare reported that Perplexity got around blocks with undeclared crawlers.

Yandex

Neuro and Alice

YandexAdditional

Controls whether the site appears in generative answers. Works only when named explicitly.

Bing

Meta tags

noarchive

Microsoft has no separate AI bot. The nocache and noarchive meta tags limit use in Bing chat answers.

04

Frequently asked questions

The main ones: GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI; ClaudeBot, Claude-SearchBot and Claude-User from Anthropic; Google-Extended from Google; PerplexityBot; Applebot-Extended; Meta-ExternalAgent; CCBot; YandexAdditional from Yandex. The full table is above: each bot has its company, class, attitude to robots.txt, a documentation link and the date it was checked.

Updated 10 October 2026

The bot list is checked against company documentation once a month. robots.txt is a request, not protection: only the server closes a site reliably.

robots.txt · AI botsHow to block AI bots in robots.txt

Generator

Basic rules

Model training

AI search and citation

User requests

What robots.txt cannot do

AI bot list

About the list

OpenAI

Anthropic

Google

Perplexity

Yandex

Bing

Frequently asked questions

which ai bots exist

how to block gptbot

how to block claudebot

does google-extended block ai overviews

why does an ai bot visit my site although it is blocked

how often is the ai bot list updated

robots.txt · AI botsHow to block AI bots in robots.txt

Generator

Basic rules

Model training

AI search and citation

User requests

What robots.txt cannot do

AI bot list

About the list

OpenAI

Anthropic

Google

Perplexity

Yandex

Bing

Frequently asked questions

which ai bots exist

how to block gptbot

how to block claudebot

does google-extended block ai overviews

why does an ai bot visit my site although it is blocked

how often is the ai bot list updated

Related tools

robots.txt generator

llms.txt generator

HTTP status codes

Token counter