AI Crawlers & Bot Access

AI companies now send more crawlers to your site than search engines ever did, and they do very different things. Some collect pages to train models, some build a search index that answers cite, and some fetch a page because a person asked right now. Block the wrong one and you vanish from ChatGPT, Claude or Perplexity answers. Allow everything without thinking and your content trains models you never agreed to. This guide is the map, current as of 2026-10-05.

The three kinds of AI bot#

Every major vendor now splits its bots by purpose:

Three rules follow, and they are the whole game:

1. Blocking a training bot does not remove you from that vendor's AI search. Blocking its search bot does.
2. Most user-triggered fetchers ignore robots.txt, because a human initiated the request. Anthropic's Claude-User is the documented exception.
3. Google is different: its AI features are built from the normal Search index, so the Googlebot rules and Search Console controls apply, not a separate AI crawler.

The 2026 bot directory#

From each vendor's own documentation, fetched 2026-10-05. Tokens are what you write after User-agent: in robots.txt.

Vendors add and rename bots several times a year (Google renamed its NotebookLM fetcher to Google-GeminiNotebook on 2026-07-16). Re-check the vendor pages in the sources before you ship a policy, and paste your file into the AI crawler checker to see the result per bot.

robots.txt is a request, not a wall#

robots.txt only works on bots that choose to honour it. In August 2025 Cloudflare published evidence that when PerplexityBot and Perplexity-User were blocked, requests continued from a generic Chrome user agent and unlisted IP addresses, and it removed Perplexity from its verified bot list. Whatever you decide, robots.txt states your policy; only a firewall or bot-management layer enforces it.

robots.txt recipes#

One rule trips up almost everyone: a crawler obeys only the most specific group that names it. Once you add User-agent: GPTBot, GPTBot ignores your User-agent: * rules. Repeat any private paths inside every named group.

Recipe 1: allow all AI bots#

If you have no rules for AI bots, they fall under User-agent: *. That is the default: maximum visibility, and your content can be used for training.

Recipe 2: allow AI search, block training#

The most common choice for businesses that want citations but not to supply training data:

User-agent: *
Disallow: /api/
Disallow: /admin/
 
# Training crawlers and opt-out tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /
 
Sitemap: https://example.com/sitemap.xml

Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) fall through to the * group. The costs: blocking Google-Extended also stops grounding in Gemini Apps, and Amazonbot and meta-externalagent serve product features as well as training, so check the vendor pages if those surfaces matter to you.

The same policy in Next.js, generated from code so the private paths stay in one place:

import type { MetadataRoute } from "next";
 
const privatePaths = ["/api/", "/admin/"];
 
const trainingBots = [
  "GPTBot",
  "ClaudeBot",
  "CCBot",
  "Google-Extended",
  "Applebot-Extended",
  "meta-externalagent",
  "Amazonbot",
];
 
export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: "*",
        disallow: privatePaths,
        // Optional: Cloudflare's Content Signals line (Next.js 16.3+ "other")
        other: { "Content-Signal": "search=yes, ai-train=no" },
      },
      { userAgent: trainingBots, disallow: "/" },
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}

Recipe 3: block all AI bots except classic search#

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: Amazonbot
User-agent: Amzn-SearchBot
Disallow: /

What it costs: no citations in ChatGPT search, Claude or Perplexity. What it doesn't do: remove you from Google's AI Overviews and AI Mode (that's Googlebot plus the controls below), stop fetchers that ignore robots.txt, or stop bots that don't identify themselves.

Google's AI features: the right control#

The Search Console setting works per property, so a URL-prefix property can scope it to one section. Note one inconsistency in Google's docs: the Search Console help page says Google-Extended also limits training of the models behind Search's generative features, while the crawler page only lists Gemini Apps and Vertex AI. Don't promise a client more than either page says. Weigh the trade-off with data first: Measuring AI Visibility shows how to see what AI features send you.

Bing and Copilot#

Bing doesn't document a separate AI crawler token. Bingbot crawls, and you control AI use with meta tags, per Bing's September 2023 announcement (Search Engine Journal reported in February 2026 that the same controls now appear in Bing's webmaster guidelines):

• <meta name="robots" content="nocache">: Copilot may use only the URL, title and snippet.
• <meta name="robots" content="noarchive">: content is not used in Copilot answers or for training Microsoft's foundation models.

The CDN layer: you may be blocking AI bots already#

Check this before anything else, because it overrides your robots.txt:

• 2025-07-01: Cloudflare began blocking AI crawlers by default for new domains and launched pay per crawl (HTTP 402 responses that let crawlers pay for access, in private beta).
• 2025-09-24: Cloudflare published the Content Signals Policy, a robots.txt extension: Content-Signal: search=yes, ai-input=…, ai-train=no. Its managed robots.txt adds search=yes, ai-train=no by default. These are stated preferences, not enforcement, and no major AI vendor has publicly committed to honouring them.
• 2026-09-15: for domains onboarding from that date, Cloudflare's AI Crawl Control defaults to three categories: Search allowed everywhere, Agent and Training blocked on pages that display ads. Crawlers that mix search and training are caught by the training block.

If your site is on Cloudflare, open AI Crawl Control in the dashboard and compare it with your intended policy. Other CDNs and WAFs (Akamai, Fastly, Vercel's firewall) have their own bot rules. A permissive robots.txt does nothing if the edge returns 403.

Verifying real bots#

User agent strings are trivially spoofed, and scrapers love to claim they are GPTBot or Googlebot. Verify before you allowlist or trust log numbers:

• Google: run a reverse DNS lookup on the IP, check it ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP. Or match against the JSON IP lists, which moved to developers.google.com/crawling/ipranges/ in February 2026 (including user-triggered-agents.json for Google-Agent).

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

• OpenAI: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json.
• Anthropic: claude.com/crawling/bots.json.
• Perplexity: perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.
• Common Crawl: index.commoncrawl.org/ccbot.json, plus reverse DNS to crawl.commoncrawl.org.

Some agents are moving to cryptographic identity: Google documented experimental Web Bot Auth signing on 2026-05-04. Anthropic notes that blocking its bots by IP also stops them reading your robots.txt, so prefer robots.txt for policy and IP checks for verification.

Reading your logs#

Your server or CDN logs show which AI bots actually visit and what they fetch. A quick first pass:

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Agent|CCBot|meta-externalagent|meta-webindexer|Amazonbot|Amzn-SearchBot|Applebot" access.log \
  | sort | uniq -c | sort -rn

How to read it:

• Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) on a page mean it is in, or being considered for, that index.
• User-triggered hits (ChatGPT-User, Claude-User, Perplexity-User) are the closest thing to demand data: a real person's question led to your page. Group them by URL to see which pages answer engines use.
• Training bots hitting thousands of URLs with no referral traffic is normal; Cloudflare measured crawl-to-referral ratios in the thousands to one for some AI vendors in mid-2025.
• Errors: 403s or 429s to bots you meant to allow usually point at a WAF rule.

Verify suspicious entries against the IP lists above before drawing conclusions.

Two things not to do#

• Don't cloak. Serving AI bots different content from what users see (stuffed keywords, hidden claims) breaks Google's spam policies, which since 2026-05-15 explicitly cover attempts to manipulate generative AI responses. The same page for everyone.
• Don't rely on llms.txt as an access or visibility control. It is an optional file for agents; Google said on 2026-06-15 that Google Search doesn't need or use it, and no major AI vendor has committed to using it for citations. The details are in the GEO Playbook.

A five-minute audit#

1. Fetch https://yoursite.com/robots.txt and paste it into the AI crawler checker.
2. Confirm every named user-agent group repeats your private-path rules.
3. Check your CDN's AI bot settings (Cloudflare: AI Crawl Control).
4. Decide your Google AI-features stance in Search Console's Search generative AI setting.
5. Grep a week of logs for AI bots and verify the top IPs.

Related: Crawling & Indexing for robots.txt fundamentals, JavaScript SEO for why most AI crawlers need server-rendered HTML, and Generative Engine Optimization for earning the citations once the bots can get in.