More productsihateimg.cv

Free tool

robots.txt for the AI era

Choose which AI bots may read your site: the ones that answer questions and link to you, and the ones that train models.

Used for the Sitemap line.

One per line. Don't list pages you want removed from Google; use a noindex tag for those.

AI search bots (ChatGPT search, Perplexity, Claude)

AI training bots (GPTBot, ClaudeBot, Google-Extended…)

Blocking search bots keeps you out of AI answers. For a product launch, allowing both usually helps people find you.

robots.txt
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

# AI search and answer engines (they cite and link to you)
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
Allow: /
Disallow: /api/
Disallow: /admin/

# AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: Amazonbot
User-agent: cohere-ai
Allow: /
Disallow: /api/
Disallow: /admin/

Sitemap: https://your-site.com/sitemap.xml

Put it at your site's root: yoursite.com/robots.txt.

Things that trip people up

  • A bot that matches a named group ignores the * group, so private paths are repeated in every group.
  • Disallow stops crawling, not indexing. To remove a page from Google, let it be crawled and add a noindex tag.
  • Blocking training bots doesn't remove you from AI search answers; blocking search bots does.

robots.txt, sitemap and llms.txt in one go

LaunchDay Kit's /launch-seo checks your site like a crawler and fixes robots.txt, sitemap, llms.txt and share tags in your code. $9 once.

Get the kit · $9 How it works

$9 for the first 30 buyers, then $29 · One-time · lifetime updates · 7-day refund