Free tool
AI companies now run separate bots for model training, for their search indexes and for fetching a page when a user asks. One robots.txt line can opt you out of training while keeping you citable, or quietly remove you from AI answers altogether. Paste your file to see exactly which bots it lets in, and which rule decided each one.
Open yoursite.com/robots.txt, copy everything and paste it below. It is checked in your browser and never sent anywhere.
291 bytes. Crawlers read the first 500 KiB and ignore the rest.
For example /blog/my-post or a full URL. Rules are case-sensitive, so /Blog and /blog differ.
AI crawlers blocked
3 / 21
Path tested
/
Classic search
Googlebot allowed, Bingbot allowed
Blocking these keeps your pages out of that vendor's model training. It does not remove you from the same vendor's AI search: that is a separate bot. Google-Extended also covers Gemini grounding, but it does not affect Google Search, AI Overviews or AI Mode. To leave those, use the generative AI setting in Search Console.
| Crawler | Result and reason |
|---|---|
| GPTBotOpenAIdocumentation for GPTBot (opens in a new tab)Honours robots.txtContent should not be used to train OpenAI's foundation models. | BlockedDisallow: / on line 10 is the longest match in User-agent: GPTBot (line 7). |
| ClaudeBotAnthropicdocumentation for ClaudeBot (opens in a new tab)Honours robots.txtPages are not collected for training Anthropic's models. | BlockedDisallow: / on line 10 is the longest match in User-agent: ClaudeBot (line 7). |
| Google-ExtendedGoogledocumentation for Google-Extended (opens in a new tab)Opt-out token, never crawlsGooglebot-crawled content is not used for Gemini training and grounding. No effect on Search ranking or inclusion. | BlockedDisallow: / on line 10 is the longest match in User-agent: Google-Extended (line 7). |
| Applebot-ExtendedAppledocumentation for Applebot-Extended (opens in a new tab)Opt-out token, never crawlsOpts out of training Apple's foundation models. Apple search inclusion and ranking are unaffected. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| meta-externalagentMetadocumentation for meta-externalagent (opens in a new tab)Honours robots.txtStops crawling for training Meta's AI models and for indexing to improve its products. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| AmazonbotAmazondocumentation for Amazonbot (opens in a new tab)Honours robots.txtStops crawling used to improve Amazon products, which may include training its AI. Amazon caches robots.txt for up to 30 days. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
These build the indexes AI assistants cite from. Blocking one removes or reduces your visibility in that assistant's answers, so block them only if you would rather not be cited at all. OpenAI notes opted-out sites can still appear as plain navigational links.
| Crawler | Result and reason |
|---|---|
| OAI-SearchBotOpenAIdocumentation for OAI-SearchBot (opens in a new tab)Honours robots.txtRemoves you from ChatGPT search results. Changes take about 24 hours. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| Claude-SearchBotAnthropicdocumentation for Claude-SearchBot (opens in a new tab)Honours robots.txtReduces your visibility in Claude's search-backed answers. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| PerplexityBotPerplexitydocumentation for PerplexityBot (opens in a new tab)Honours robots.txtRemoves you from Perplexity search results. Not used for training. | AllowedUses User-agent: PerplexityBot (line 12), which has no rule matching this path. |
| ApplebotAppledocumentation for Applebot (opens in a new tab)Honours robots.txtRemoves you from Spotlight, Siri and Safari suggestions. Its data may also train Apple models. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| meta-webindexerMetadocumentation for meta-webindexer (opens in a new tab)Honours robots.txtReduces your presence in Meta's AI search results. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| Amzn-SearchBotAmazondocumentation for Amzn-SearchBot (opens in a new tab)Honours robots.txtRemoves you from Amazon search experiences, including Alexa. Not used for training. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
These fetch a page because a person asked an assistant to, not to build an index. Most vendors say robots.txt may not apply to them; Anthropic's Claude-User is the documented exception. If you really need to stop them, use your CDN or firewall: robots.txt is a request, not enforcement.
| Crawler | Result and reason |
|---|---|
| ChatGPT-UserOpenAIdocumentation for ChatGPT-User (opens in a new tab)Generally ignores robots.txtAsks ChatGPT not to open your pages when a user requests them, but OpenAI says the rules may not apply. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| Claude-UserAnthropicdocumentation for Claude-User (opens in a new tab)Honours robots.txtClaude will not fetch your pages when a user asks it to. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| Perplexity-UserPerplexitydocumentation for Perplexity-User (opens in a new tab)Generally ignores robots.txtLittle effect: Perplexity says this fetcher generally ignores robots.txt. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| Google-AgentGoogledocumentation for Google-Agent (opens in a new tab)Generally ignores robots.txtLittle effect: AI agents browsing for a user on Google infrastructure ignore robots.txt. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| meta-externalfetcherMetadocumentation for meta-externalfetcher (opens in a new tab)May bypass robots.txtPartial: Meta says user-requested fetches may bypass robots.txt. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| Amzn-UserAmazondocumentation for Amzn-User (opens in a new tab)May bypass robots.txtPartial: live answers such as Alexa's may bypass some directives. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
Single-purpose crawlers for ads review, research or crawls you request yourself. They are rarely worth blocking, but they show up in logs, so it helps to know what they are.
| Crawler | Result and reason |
|---|---|
| OAI-AdsBotOpenAIdocumentation for OAI-AdsBot (opens in a new tab)Honours robots.txtStops validation of pages submitted as ChatGPT ads. Not used for training. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| Google-CloudVertexBotGoogledocumentation for Google-CloudVertexBot (opens in a new tab)Honours robots.txtStops crawls a site owner requested for Vertex AI Agents. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| GoogleOtherGoogledocumentation for GoogleOther (opens in a new tab)Honours robots.txtStops Google's internal research and one-off product crawls. No effect on Search. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
Blocking Googlebot removes you from Google Search, and with it AI Overviews and AI Mode, which draw on the Search index. Blocking Bingbot removes you from Bing. Neither is a way to opt out of AI alone.
| Crawler | Result and reason |
|---|---|
| GooglebotGoogledocumentation for Googlebot (opens in a new tab)Honours robots.txtRemoves your pages from Google Search, AI Overviews and AI Mode. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
| BingbotMicrosoftdocumentation for Bingbot (opens in a new tab)Honours robots.txtRemoves your pages from Bing Search. | AllowedFalls back to User-agent: * (line 2), which has no rule matching this path. |
Starting points, not drop-in files: merge them with the rules you already have, then load the result above and check it before you publish.
Stay citable in ChatGPT, Claude, Perplexity and AI Overviews while opting out of model training. Google-Extended also turns off Gemini grounding.
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot Disallow: / User-agent: * Allow: /
Opts out of every AI crawler that honours robots.txt. Googlebot and Bingbot stay allowed, so you can still appear in AI Overviews; leaving those is a Search Console setting.
User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: meta-webindexer User-agent: Amazonbot User-agent: Amzn-SearchBot Disallow: / User-agent: * Allow: /
Leaves the site open but keeps a members area out of AI search indexes. Remember robots.txt is public: it is not access control.
User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Disallow: /members/ User-agent: * Disallow: /admin/
The open default. An empty Disallow allows all; listing the sitemap helps every crawler find your URLs.
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
* group if none does, never both. Within that group the longest matching path wins, and Allow wins a tie.User-agent: *. In the example, PerplexityBot can crawl /admin/ because its group only blocks drafts. This is the most common robots.txt surprise.For the full picture of which bots to allow and why, read the AI crawlers guide.