Robots.txt for AI crawlers: which bots to allow (2026)

If an AI assistant's crawler can't read your website, that assistant can't recommend you from it. Plenty of small business sites block AI bots without the owner ever knowing. The culprit is usually an old robots.txt rule, a security plugin, or bot protection your web host switched on. This lesson covers which bots matter, what each one does, and how to let the right ones in.

What robots.txt is

robots.txt is a plain text file at the root of your website, for example https://www.example.com/robots.txt. It tells automated crawlers which parts of the site they may visit. The format is an internet standard, RFC 9309, published in September 2022. A few of its rules are worth knowing:

The file must be at the top level of the site and named exactly robots.txt.
Rules are grouped by user agent, the name a crawler identifies itself with. Crawlers match their name without caring about upper or lower case, and use the * group if no group names them.
When several rules match a page, the most specific one (the longest match) wins.
robots.txt is a polite request. The standard says these rules "are not a form of access authorization." Well-behaved crawlers follow them, but they do nothing to protect private content.

Three kinds of AI bots

AI companies now run several bots, each with its own job. Lumping them together is the mistake we see most.

Search crawlers build the index an assistant searches when it answers questions. Block one and you vanish from that assistant's answers.
Training crawlers collect content that may be used to train future AI models. OpenAI and Google say blocking their training controls doesn't remove you from their search results.
User-triggered fetchers visit a page because a person asked the assistant to look at it. Some companies say robots.txt may not apply to these, because a human started the request.

The bots, one by one

Every row below comes from the company's own documentation.

User agent	Company	Kind	What it controls	If you block it
`Googlebot`	Google	Search	Google Search, including AI Overviews and AI Mode	You drop out of Google Search, AI features included. Don't.
`Google-Extended`	Google	Training control	Use of your content to train future Gemini models and to ground answers in Gemini Apps and Vertex AI	No effect on Google Search. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
`Bingbot`	Microsoft	Search	Bing results, which Copilot uses	You drop out of Bing and Copilot answers.
`OAI-SearchBot`	OpenAI	Search	ChatGPT search answers	You won't be "shown in ChatGPT search answers" (navigational links may still appear).
`GPTBot`	OpenAI	Training	Content that "may be used in training" OpenAI's models	No effect on ChatGPT search; OpenAI treats each bot's setting independently.
`ChatGPT-User`	OpenAI	User-triggered	Pages a ChatGPT user asks it to visit	OpenAI says robots.txt rules "may not apply." Not used to decide what appears in search.
`PerplexityBot`	Perplexity	Search	Perplexity search results	You won't be surfaced and linked in Perplexity. Perplexity says it isn't used to train AI foundation models.
`Perplexity-User`	Perplexity	User-triggered	Pages fetched to answer a user's question	Perplexity says it "generally ignores robots.txt rules."
`ClaudeBot`	Anthropic	Training	Content that may be used to train Anthropic's models	Signals that your future content should be excluded from training.
`Claude-SearchBot`	Anthropic	Search	Search results in Claude	Anthropic won't index your content for search.
`Claude-User`	Anthropic	User-triggered	Pages fetched when a Claude user asks a question	Anthropic won't retrieve your content for those questions.

Two details catch people out:

Google-Extended isn't a separate crawler. Google says it "doesn't have a separate HTTP request user agent string." Googlebot does the crawling; the Google-Extended rule only controls how the content may be used.
Changes aren't instant. OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust.

A robots.txt that lets AI search find you

Most local businesses want to be found everywhere people search. This example lets every search crawler in and leaves the training question to you. Swap example.com for your own domain.

# Search crawlers: allow (these decide whether you appear in answers)
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training crawlers: your choice. Delete this block to allow training.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Allowing training crawlers is a business call more than an SEO one. OpenAI and Google say explicitly that blocking their training controls doesn't affect search, and Anthropic runs a separate bot for search. Letting them in may mean future models know more about your business. Nobody has published evidence either way on what that's worth to a local business.

If your file has a User-agent: * group with Disallow: /, every bot not named elsewhere is shut out. That's a common leftover from a site's "coming soon" days.

Other ways sites block AI by accident

robots.txt isn't the only thing standing at the door:

Firewalls and CDNs. Google's AI features guidance asks you to make sure crawling is allowed "by any CDN or hosting infrastructure," not just robots.txt. OpenAI asks sites to allow traffic from its published IP addresses, and Perplexity publishes its IP ranges too. If your host or security plugin has a "block AI bots" switch, find out exactly what it blocks.
Snippet controls. Google's nosnippet, data-nosnippet and max-snippet settings limit what can be shown from a page, including in AI features. Bing's NOARCHIVE keeps a page out of Bing's AI chat answers (Bing Chat is now Copilot), and NOCACHE limits them to the URL, title and snippet. Make sure none are set on pages you want quoted.
Google's Search generative AI control. Search Console now has a setting (Settings > Search generative AI) to exclude your site from AI Overviews, AI Mode and AI features in Discover. Google says it rolled this out to all sites as of August 31, 2026, that "include" is the default, and that excluding isn't used as a ranking signal elsewhere in Search. If you want to be found, leave it on include.

llms.txt: a reality check

llms.txt is a proposed file, suggested by Jeremy Howard in September 2024, that gives AI tools a short Markdown summary of a site. For software documentation sites, it's a sensible idea. For AI search, this is where things stand as of October 2026:

Google says Google Search ignores llms.txt and similar files: adding one "will neither harm nor help" your visibility. It says you can keep one for other services that use it.
OpenAI, Perplexity and Anthropic don't mention llms.txt in the crawler documentation we checked.
The proposal's own site doesn't claim that any AI search engine uses it.

So it's harmless, and it's not a priority. Fix robots.txt, your firewall settings and your business listings first.

How to check your own site

Open yourdomain.com/robots.txt in a browser. If it's missing, crawlers assume everything is allowed.
Look for Disallow: / under * or under any search crawler named in the table.
In Google Search Console, use the URL Inspection tool on your homepage to confirm Google can crawl and index it.
In Bing Webmaster Tools, do the same with Bing's URL inspection.
Ask your web host or developer whether any "bot protection" or "block AI" setting is on, and what it blocks.

Do this now

Open yourdomain.com/robots.txt and confirm nothing blocks Googlebot, Bingbot, OAI-SearchBot, PerplexityBot or Claude-SearchBot.
Decide on purpose whether to allow or block training crawlers (GPTBot, ClaudeBot, Google-Extended).
Remove any leftover "User-agent: *" plus "Disallow: /" rule.
Ask your host or developer whether a firewall, CDN or security plugin blocks AI crawlers.
Check that the Search generative AI setting in Search Console is on "include."
Run URL Inspection on your homepage in both Google Search Console and Bing Webmaster Tools.

FAQ

If I block GPTBot, will I disappear from ChatGPT?

No. OpenAI says each bot's setting is independent: you can allow OAI-SearchBot for search while blocking GPTBot for training. Only blocking OAI-SearchBot keeps you out of ChatGPT search answers.

Does blocking Google-Extended hurt my Google rankings?

No. Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal. It controls Gemini training and grounding in some other Google products.

Should I add an llms.txt file?

It's optional. Google says Search ignores it, and the crawler docs from OpenAI, Perplexity and Anthropic we checked don't mention it. It does no harm, but fixing robots.txt and your listings matters far more.

Sources

RFC 9309: Robots Exclusion Protocol, IETF / RFC Editor (checked Oct 2026)
Overview of OpenAI crawlers, OpenAI (checked Oct 2026)
Perplexity crawlers, Perplexity Docs (checked Oct 2026)
Does Anthropic crawl data from the web?, Anthropic (Claude Help Center) (checked Oct 2026)
Google's common crawlers (Google-Extended), Google Search Central (checked Oct 2026)
AI features and your website, Google Search Central (checked Oct 2026)
Search generative AI control, Search Console Help (checked Oct 2026)
Optimizing for generative AI features: mythbusting, Google Search Central (checked Oct 2026)
New options to control usage of content in Bing Chat (NOCACHE, NOARCHIVE), Bing Webmaster Blog (checked Oct 2026)
The /llms.txt file proposal, llmstxt.org (checked Oct 2026)

Want this done for you? Start free

You'll learn