AI Crawlers, robots.txt and llms.txt: What to Allow on Your Site and Why

Your server logs show bots with names like GPTBot and ClaudeBot, and the advice online ranges from "block everything before they steal your content" to "allow everything or become invisible." Both miss the actual structure of the problem: these bots do three different jobs, and each vendor deliberately gives you a separate switch for each job.

By the end of this guide you will know which user agents exist as of 2026, what each one does, a defensible allow/disallow decision for your site, and where llms.txt honestly fits. It assumes you can edit robots.txt at your site root; for related indexing topics see our guides.

The three jobs, and the tokens that control them

Every major AI vendor now splits its crawlers by purpose. The tokens below come from each vendor's own documentation as of 2026:

The critical row pairs are the first two of each vendor: blocking the training bot does not remove you from that vendor's search product. OpenAI states this explicitly, and Anthropic's three-way split works the same way. The blanket Disallow: / rules that circulate in blocklists usually block the search bots too — the ones that send you readers.

A second distinction worth knowing: user-triggered fetchers are not crawlers. They fetch a single page because a person asked for it. Vendors differ on whether robots.txt applies to them, so treat those tokens as advisory and enforce at your WAF or server if you truly need a hard block.

A sample robots.txt policy

This site's own posture — allow the bots that cite, decline the ones that train — looks like this. Yours should reflect your own decision; there is no universally correct answer.

# Keep classic crawlers untouched: no Googlebot rules needed at all.

# Training bots: opt out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# AI search bots: allow (these are how AI answers cite sources)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Three rules of thumb for your own version:

1. Never write a rule for Googlebot or Bingbot unless you have a specific reason. The whole point of this file is surgical control of AI tokens; the classic search crawlers should see the same Allow everyone else gets.
2. User-agent blocks match by token substring, not full UA string — ClaudeBot matches ClaudeBot/1.0 regardless of version, which is what you want since vendors bump versions.
3. Test after deploying: Search Console's robots.txt tester validates syntax against Google crawlers; for AI tokens, the test is simply checking your access logs over the following weeks.

What llms.txt actually is (and is not)

llms.txt is a community proposal (llmstxt.org, first published 2024) for a Markdown file at /llms.txt that gives AI agents a curated map of your site: an H1, a summary blockquote, then H2-delimited link lists pointing at LLM-friendly versions of pages. The proposal also suggests serving .md versions of pages and linking them via rel="alternate" type="text/markdown".

Honest assessment as of 2026: adoption is real but unproven as a ranking or citation lever. Thousands of sites publish one, documentation platforms generate it automatically, Chrome's Lighthouse audits for it in agentic checks, and the AI labs publish their own — but no major vendor has documented that its crawlers consult llms.txt when indexing or answering.

This site emits one anyway — /llms.txt, /llms-full.txt, a JSON index, and per-guide Markdown at index.md — because the cost is near zero and the file doubles as a tidy machine-readable sitemap. Treat it the same way: cheap insurance and good hygiene, not a guarantee.

Verifying that bots obey — and what to do when they don't

robots.txt is a contract of good behaviour, not a gate. Any scraper can copy a legitimate user-agent string, and some fetchers — Perplexity-User is documented as generally ignoring robots.txt — are outside the contract by design. So verification has two layers:

• Logs first. Watch your access logs or CDN analytics for the tokens you blocked. Well-behaved vendors stop fetching within about a day of a policy change (OpenAI documents roughly 24 hours for search changes to take effect).
• Enforce only where it matters. If a blocked bot keeps coming — or an imposter uses the token — the WAF/server layer is the real control. Each vendor publishes its crawler IP ranges as JSON (OpenAI, Anthropic, and Perplexity all do), so an allow/block rule can check both user-agent and source IP, which defeats spoofed UAs.

One more honest caveat: robots.txt files are public. Listing a directory under Disallow announces that it exists — fine for telling bots to stay out, useless for hiding anything. Private content belongs behind authentication, not robots rules.

Deciding your own policy

1. List the bots you actually see in your access logs (server or CDN analytics) — policy should cover reality, not a copied blocklist.
2. Decide the training question separately from the visibility question: training bots take; search bots send. There is a defensible case for blocking training and allowing search, for allowing everything, or for blocking everything on a paywalled/private corpus.
3. Write one User-agent block per token from the table above — avoid wildcards like Claude*, which is not a valid robots.txt token anyway.
4. Deploy, then check logs in 2-4 weeks to confirm the bots behave as documented; escalate to WAF rules only for tokens that ignore your file.
5. Revisit the list quarterly — vendors add tokens and occasionally split agents, as OpenAI did when it separated OAI-SearchBot from GPTBot.

When this whole exercise is unnecessary: if your site has no public content worth citing (a login-only app, an intranet), a plain Disallow: / under User-agent: * remains the right answer and none of the nuance above matters.

— Editorial team. Facts current as of 2026; we revise guides when the ground shifts.