what happens if i block all ai bots

Along with training you close the site to AI search: ChatGPT, Perplexity, Claude and Yandex Neuro stop showing it in answers with links. For sites that need traffic this is usually a bad deal. On top of that, bots that fetch pages for users often state they may not follow robots.txt, so the file cannot shut you off completely anyway.

how to block yandex neuro in robots.txt

Add a separate group “User-agent: YandexAdditional” with “Disallow: /”. The robot reads only rules where it is named: a block for “*” does not apply to it, and that is a common mistake. Ordinary Yandex search is not affected. The generator adds the group on its own line, and the checker warns when YandexAdditional is closed only by the asterisk.

do ai bots respect robots.txt

Training and search bots of major companies do, according to their documentation. Bots that open pages at a user's request often do not: OpenAI says so about ChatGPT-User, and Perplexity, Meta, Amazon and Google say the same. Only rules on the server or in a CDN close the site to them reliably; for them robots.txt stays a request.

how to test robots.txt online

Paste the file into the “Check” tab, enter a path and pick a robot — we show whether crawling is allowed and which rule matched, with its line number and group. The check follows Google and Yandex rules: the longest rule wins, and on a tie Allow wins. Below is a list of file errors: obsolete Host, Crawl-delay, paths without a slash, non-ASCII paths.

is the host directive still needed

No. Yandex no longer supports the Host directive: the main mirror is set with a 301 redirect and declared in Yandex Webmaster. Google never understood it. The generator does not add Host, and the checker flags it as obsolete in a ready file — the line can be removed safely. The same goes for Crawl-delay, which Google ignores.

Code & data · robots.txtrobots.txt generator for AI bots and Yandex

Build a robots.txt that lets search engines in, decides on AI bots class by class and talks to Yandex correctly. Paste a ready file — we will check who it actually lets in.

The file is built in your browser — neither rules nor addresses go to a server

01Generator 02About robots.txt 03Questions 04Related

Generator

Basic rules

Block sections · one path per lineSitemap · one address per line

Clean-param for Yandex · parameters · path

Presets

Model training

GPTBot, ClaudeBot, Google-Extended, CCBot…

Blocking affects neither search nor answers with links.

GPTBot

OpenAI

follows robots.txtdocumentation

ClaudeBot

Anthropic

follows robots.txtdocumentation

Google-Extended · token

Google

follows robots.txtdocumentation

Applebot-Extended · token

Apple

follows robots.txtdocumentation

CCBot

Common Crawl

follows robots.txtdocumentation

Meta-ExternalAgent

AI search and citation

OAI-SearchBot, Claude-SearchBot, PerplexityBot, YandexAdditional…

Blocking removes the site from answers with links — that is lost traffic.

User requests

ChatGPT-User, Claude-User, Perplexity-User…

Many of them may ignore robots.txt; only the server is reliable.

Safety net: add the old names anthropic-ai, Claude-Web, FacebookBot — current documentation no longer lists them, but old records still use them131 more bots from the community — not split into classes, and some of them are search bots. List ai.robots.txt · MIT

robots.txt · 27 lines

User-agent: *
Disallow: /admin/
Disallow: /search
# AI: model training — blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: cohere-training-data-crawler
User-agent: Timpibot
Disallow: /
# Yandex: Neuro — allowed except the general blocks (it does not read *)
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
Disallow: /admin/
Disallow: /search
Clean-param: utm_source&utm_medium&utm_campaign /
Sitemap: https://example.ru/sitemap.xml

What robots.txt cannot do

Google AI Overviews and Bing chat run on the ordinary search robots — limit them with meta tags in <head>.

Google AI Overviews and AI Mode

<meta name="robots" content="nosnippet"> or max-snippet:0

Bing chat answers

<meta name="robots" content="noarchive, nocache">

About robots.txt

robots.txt tells robots which parts of a site they may crawl. With the arrival of AI it got a new job: deciding whether a company may train a model on your texts and show them in chatbot answers. AI bots come in three kinds, and blocking each has a different effect — the generator shows it before you save the file.

Training

GPTBot, ClaudeBot

Disallow: /

Blocking stops the site's texts from being used to train models and affects neither search nor citation.

AI search

OAI-SearchBot, PerplexityBot

traffic

These bots bring the site into answers with links. Blocking removes you from such answers.

User requests

ChatGPT-User

may ignore

Bots that open a page at a person's request. Many state outright that they may not follow robots.txt.

Google-Extended

A token, not a robot

no effect on Search

Blocks Gemini training and grounding in Gemini and Vertex AI, but does not affect Google Search or AI Overviews — those run on the ordinary Googlebot.

YandexAdditional

Yandex

by name only

Stops the site from appearing in Neuro and Alice answers. Works only when named explicitly: it does not read rules for “*”.

No Host

Obsolete

Host:

Yandex dropped the Host directive long ago — the main mirror is set with a redirect. The generator does not add it.

Frequently asked questions

Block the “training” class in robots.txt: GPTBot, ClaudeBot, Google-Extended, CCBot, Meta-ExternalAgent and others. The generator puts them in one group with “Disallow: /” — that is the “Don't train, but cite” preset. Such a block affects neither search nor answers that link to your site: other bots handle those, and it leaves them alone.

Updated 10 October 2026

The bot list is checked against company documentation once a month. robots.txt is a request, not protection: only the server closes a site reliably.

Code & data · robots.txtrobots.txt generator for AI bots and Yandex

Generator

Basic rules

Model training

AI search and citation

User requests

What robots.txt cannot do

About robots.txt

Training

AI search

User requests

Google-Extended

YandexAdditional

No Host

Frequently asked questions

how to stop ai bots training on my site

what happens if i block all ai bots

how to block yandex neuro in robots.txt

do ai bots respect robots.txt

how to test robots.txt online

is the host directive still needed

Related tools

HTTP status codes

URL cleaner

Clean up AI text

Token counter