The Definitive Technical Guide
Hosting for AI Crawlers: Configuring Your Server for GPTBot, ClaudeBot & AI Search Visibility
The complete bot directory, robots.txt rules, and server-level config for the AI traffic already hitting your logs
📋 What’s in this guide
- Three Kinds of AI Bot Traffic
- The Complete AI Crawler Directory
- Should You Block or Allow Them?
- Writing robots.txt Rules for AI Bots
- Server-Level Blocking (When robots.txt Isn’t Enough)
- llms.txt: What It Is and Isn’t
- Making Your Content AI-Citation-Friendly
- Monitoring AI Bot Traffic on Your Server
- Complete AI Crawler Configuration Checklist
- Frequently Asked Questions
If you’ve looked at your server logs recently and noticed a wave of unfamiliar user-agents — GPTBot, ClaudeBot, PerplexityBot, Bytespider, and a dozen others — you’re not imagining it. AI companies now operate more crawlers hitting the open web than most site owners have ever had to think about configuring for, and they don’t all behave the same way, want the same thing, or even respect the same rules.
This guide is the practical, server-configuration side of that problem: which bots exist, what each one actually does, how to control them through robots.txt and your server config, and what’s genuinely useful right now versus what’s still speculative. We’ve covered the traffic and behavior patterns of this shift from a data perspective elsewhere — see our AI agent traffic research and bot traffic statistics for the numbers. This page is about what to actually do about it on your own server.
1. Three Kinds of AI Bot Traffic
The single most useful thing to understand before writing a single robots.txt rule is that “AI bot” isn’t one category. Every major AI company now operates crawlers that fall into three functionally distinct groups, and the right decision for each group is often completely different.
1. Training Crawlers
These bots (GPTBot, ClaudeBot, Google-Extended, Bytespider, CCBot) crawl your site to collect content used in training or fine-tuning AI models. Once your content is in a training set, it’s baked into the model’s general knowledge — not retrieved live, not attributed at the point of use, and not something you can revoke after the fact. This is the category where the “should I block this” question is most genuinely contested.
2. Search & Answer Crawlers
These bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Google-Extended’s search-adjacent uses) crawl to build a retrieval index that powers real-time, cited answers — the AI equivalent of a search engine’s crawler. Blocking these doesn’t protect your content from being “used to train AI”; it just removes you from that engine’s ability to cite and link to you when someone asks it a question your page could answer.
3. User-Triggered Fetchers
These bots (ChatGPT-User, Claude-User, Perplexity-User) don’t crawl proactively at all — they fetch a specific page in real time because a human user asked a question that led the assistant to that exact URL. Several of these explicitly state they may not respect robots.txt, since a person, not an automated process, requested the fetch.
Blocking a training crawler protects your content from being absorbed into a model. Blocking a search/answer crawler removes you from that assistant’s citations — the AI equivalent of de-indexing yourself from a search engine. Treating both decisions the same way, in either direction, means either giving up training-data control you might actually want, or accidentally opting out of exactly the “AI search visibility” traffic most site owners are trying to capture.
2. The Complete AI Crawler Directory
The table below covers the crawlers publishers most commonly need to configure for, with the category from Section 1, the operator’s own stated purpose, and whether robots.txt is confirmed to control it.
| User-Agent Token | Operator | Category | Respects robots.txt? |
|---|---|---|---|
| GPTBot | OpenAI | Training | Yes |
| OAI-SearchBot | OpenAI | Search / ChatGPT search | Yes |
| OAI-AdsBot | OpenAI | Ad landing page validation | Yes |
| ChatGPT-User | OpenAI | User-triggered fetch | May not apply (user-initiated) |
| ClaudeBot | Anthropic | Training | Yes |
| Claude-SearchBot | Anthropic | Search / answer quality | Yes |
| Claude-User | Anthropic | User-triggered fetch | Yes, but user-initiated |
| Google-Extended | Gemini training / grounding | Yes (no Search impact either way) | |
| Google-CloudVertexBot | Vertex AI Agent building (owner-requested) | Yes | |
| GoogleOther | Internal R&D, one-off crawls | Yes | |
| PerplexityBot | Perplexity | Search results indexing | Yes |
| Perplexity-User | Perplexity | User-triggered fetch | Generally ignores robots.txt (user-initiated) |
| Bytespider | ByteDance (TikTok) | Training | Inconsistent compliance reported |
| CCBot | Common Crawl | Open dataset (widely reused for training) | Yes |
| Amazonbot | Amazon | Alexa / product search & training | Yes |
| Applebot-Extended | Apple | Apple Intelligence training | Yes |
| meta-externalagent | Meta | Training & search indexing | Yes |
This list changes — new bots appear and existing ones get renamed or split (Anthropic’s own three-bot framework above is itself a 2025 restructuring of what used to just be “ClaudeBot”). Treat this table as a snapshot, and verify current tokens against each operator’s own documentation before finalizing a policy you intend to leave in place for years.
A user-agent string is just a text header any client can send — including a scraper impersonating GPTBot to bypass a block. OpenAI, Anthropic, and Perplexity all publish official IP address ranges (as static JSON files at their own domains) that their real crawlers originate from. For anything beyond a basic robots.txt courtesy request, verify traffic against the published IP list before trusting the user-agent header alone, especially if you’re allowing a bot through a firewall rule.
3. Should You Block or Allow Them?
There’s no universally correct answer here — it depends on what you’re optimizing for, and the honest trade-offs are different for each category from Section 1.
The Case for Allowing Search & Answer Crawlers
If your goal is AI search visibility — showing up as a cited source when someone asks ChatGPT, Claude, or Perplexity a question your content answers — blocking OAI-SearchBot, Claude-SearchBot, or PerplexityBot is directly counterproductive. These are the closest AI-era equivalent to Googlebot for organic search: block them and you’re invisible to that channel entirely, not protected from it.
The Case for Blocking (or Rate-Limiting) Training Crawlers
Blocking GPTBot, ClaudeBot, or Google-Extended doesn’t remove you from AI search results — those are separate bots, per Section 2 — it only opts your content out of being used as training data for future model versions. Publishers with genuine concerns about content reuse, competitive risk (a model trained partly on your proprietary analysis, later summarizing it without attribution), or contractual restrictions on their content most commonly restrict this category specifically.
The Case for Allowing Everything
Some site owners take the position that broader AI visibility — being represented accurately in models people already use daily — outweighs the training-data concern, particularly for informational or reference content where being the source an AI model draws from is itself a form of distribution and brand exposure. There’s no wrong answer here; it’s a judgment call about your specific content and business model, not a technical requirement.
Independent of the content-policy question, AI crawlers — particularly aggressive or poorly-throttled ones — consume real server resources: CPU, bandwidth, and database queries on dynamic pages. On budget shared hosting with limited resource allocation, a spike in AI crawler traffic can measurably affect performance for actual human visitors. This is a legitimate, purely operational reason some site owners rate-limit or block certain bots regardless of where they land on the content-policy question above.
4. Writing robots.txt Rules for AI Bots
robots.txt rules for AI bots work exactly like any other robots.txt directive — a User-agent line naming the bot, followed by Allow or Disallow rules. Each bot gets its own block; there’s no wildcard specifically for “all AI bots” as a category, so you have to name each one you want to control individually.
Block All Training Crawlers, Allow Search/Answer Crawlers
This is the most common configuration for sites that want AI search visibility without contributing to model training — block the training-specific bots by name, and either allow or simply don’t mention the search-oriented ones (an unmentioned bot is allowed by default).
# Training crawlers -- opt out of model training User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: / # Search / answer crawlers -- explicitly allowed User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: /
Block Everything AI-Related
For sites that want no AI crawler access at all — training or search — block every known token individually. Remember this doesn’t affect the user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User), several of which don’t reliably honor robots.txt since a human explicitly requested that single-page fetch.
User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: /
Rate-Limit Instead of Blocking
Several AI crawlers, including ClaudeBot, honor the non-standard but widely-supported Crawl-delay directive — useful if your concern is server load rather than content policy.
User-agent: ClaudeBot Crawl-delay: 5
5. Server-Level Blocking (robots.txt Isn’t Enough)
robots.txt is a request, not an enforcement mechanism — it works because reputable operators choose to honor it, not because your server actually stops non-compliant traffic. For bots with inconsistent compliance, or if you want a harder guarantee, block at the server level using the same .htaccess user-agent matching covered in our complete .htaccess guide.
<IfModule mod_rewrite.c> RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (GPTBot|CCBot|Bytespider) [NC] RewriteRule ^ - [F,L] </IfModule>
This returns a 403 Forbidden to any request whose user-agent string contains one of the named tokens — effective against compliant and non-compliant bots alike, since it’s enforced by your server rather than requested politely. The trade-off is maintenance: unlike robots.txt (a single text file any operator can check), user-agent matching has to be updated manually as bots rename themselves or new ones appear.
If your site sits behind Cloudflare or a similar CDN/WAF, most now offer a managed, regularly-updated AI bot category you can block or allow with a toggle, rather than hand-maintaining an .htaccess regex list yourself. This is generally the lower-maintenance option for sites without a specific reason to control blocking at the origin server level, since the provider updates the underlying bot list as the landscape changes.
6. llms.txt: What It Is and Isn’t
llms.txt is a proposed convention — a plain-markdown file placed at your site’s root, listing curated links and summaries meant to be easier for an AI agent to parse than a full HTML page. It was proposed in 2024, and by 2026 has seen real adoption: OpenAI, Anthropic, and Google have all published llms.txt files for their own developer documentation, and it’s become common enough that some CMS platforms generate one automatically.
What It Actually Does Right Now
Here’s the honest, unhyped state of it, and as of mid-2026 it’s more settled than it used to be. Google’s own AI-optimization documentation was updated in June 2026 to state directly that creators “don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search,” and that an llms.txt file “won’t harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them.” That same month, Google’s John Mueller called llms.txt adoption “purely speculative for now” in a public Reddit response, pointing out the file “has existed for years, yet none of the AI systems use it.”
That’s a direct statement about Google Search specifically — it doesn’t settle the question for other AI assistants. There is still no confirmed, official statement from OpenAI, Anthropic, or Perplexity that their production crawlers or assistants read and use llms.txt files when crawling the web or answering user questions. The fact that OpenAI, Anthropic, and Google have all published their own llms.txt files for their own developer documentation shows institutional awareness of the convention — it doesn’t confirm their crawlers consume other sites’ llms.txt files as part of training or retrieval.
If you’ve seen llms.txt described as “essential for AI search visibility” or “how you get cited by ChatGPT,” treat that claim skeptically. Google has now said outright that it ignores the file for Search. For the other assistants, it remains a plausible-future-value addition rather than a confirmed one. The bots that demonstrably do affect your AI visibility are the search/answer crawlers in Section 2, which respond to your normal HTML content, not a separate manifest file.
The file is still low-cost to add (a few minutes of work, and it can’t hurt anything per Google’s own statement), so there’s no harm in adding a basic one as a forward-looking hedge for assistants other than Google’s. Just don’t reallocate effort away from the things confirmed to matter (clean HTML, fast TTFB for real-time fetches, and allowing the actual search/answer crawlers) in order to build an elaborate one, and don’t expect it to move your Google visibility at all.
7. Making Your Content AI-Citation-Friendly
Beyond the allow/block decision, there’s a hosting-adjacent factor that’s easy to miss: user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) retrieve your page in real time, during an active conversation, while a person is waiting for an answer. Unlike a search engine’s crawl-and-index cycle, there’s no caching layer smoothing over a slow response — your server’s TTFB directly determines whether that fetch completes before the assistant’s own response-time budget runs out.
- Fast TTFB matters more here than almost anywhere else — a slow, uncached page risks the fetch timing out entirely, meaning the assistant simply can’t use your page as a source for that specific answer, regardless of how relevant your content is.
- Clean, semantic HTML parses more reliably than content buried in heavy client-side JavaScript rendering, which some crawlers may not execute at all.
- Clear, direct-answer structure — a specific paragraph that directly answers a specific question — is what gets extracted and quoted; walls of unstructured text are harder for any extraction system, human or AI, to cite precisely.
- Structured data (schema.org markup) gives crawlers unambiguous, machine-readable facts (author, publish date, organization) that reduce the odds of misattribution.
Notice that every item on this list is also just good practice for human readers and for traditional SEO — there’s no separate “AI-only” content strategy required. The infrastructure requirement is the new part: hosting fast enough that a real-time fetch completes reliably.
8. Monitoring AI Bot Traffic on Your Server
Before deciding on a blocking policy, it’s worth actually looking at what’s hitting your server. Most hosting control panels and log analyzers let you filter access logs by user-agent string.
# Count hits per AI bot from a standard access log grep -oE "GPTBot|ClaudeBot|Google-Extended|PerplexityBot|Bytespider|CCBot|Amazonbot|meta-externalagent" access.log | sort | uniq -c | sort -rn
This gives you a rough sense of which bots are actually active on your site and how frequently — useful context before writing a blocking policy based on a bot’s reputation alone, rather than its actual footprint on your server. On managed hosting without raw log access, most control panels (cPanel’s “Raw Access Logs,” or your host’s analytics dashboard) provide the same data in a more accessible format.
- Pull a week of raw access logs and filter for known AI user-agent tokens
- Identify which bots are generating meaningful request volume vs. occasional/negligible traffic
- Cross-reference high-volume bots against the published IP ranges in Section 2 to rule out impersonation
- Decide your policy per bot category (Section 3), not as a blanket allow/block
- Implement via robots.txt first; escalate to server-level blocking only for confirmed non-compliant bots
9. Complete AI Crawler Configuration Checklist
- Reviewed which AI bots are actually hitting your server via access logs
- Decided a policy per category: training crawlers, search/answer crawlers, user-triggered fetchers
- robots.txt updated with explicit rules for each named bot in your policy
- Server-level (.htaccess or CDN/WAF) blocking configured for any non-compliant bots you want hard-blocked
- Confirmed high-volume bot traffic against official published IP ranges
- TTFB verified fast enough for real-time user-triggered fetches to complete reliably
- Structured data (schema.org) in place for key pages to reduce misattribution risk
- Basic llms.txt added as a low-cost forward-looking hedge (optional, not a confirmed requirement)
- Bot policy documented somewhere so it survives a future robots.txt rewrite by mistake
10. Frequently Asked Questions
Will blocking GPTBot remove my site from ChatGPT entirely?
No — GPTBot is specifically OpenAI’s training crawler. Blocking it opts your content out of future model training, but ChatGPT’s search/browsing features use OAI-SearchBot and the user-triggered ChatGPT-User, which are separate bots with their own robots.txt entries. Your content can still be findable and citable in ChatGPT’s search features even with GPTBot fully blocked.
Does robots.txt retroactively remove my content from an AI model that already trained on it?
No. robots.txt only affects future crawling. If a training crawler already collected your content before you added a Disallow rule, that data may already be part of a trained model, and there’s no mechanism to retroactively remove it from a model that’s already been trained. This is a meaningful difference from search engine indexing, where removing a page from crawling generally does eventually remove it from the index.
Can I block AI crawlers without hurting my regular SEO?
Yes — AI crawlers and Googlebot/Bingbot are entirely separate user-agents with independent robots.txt rules. Blocking GPTBot, ClaudeBot, or similar has zero effect on Googlebot’s crawling or your traditional search rankings, since they’re unrelated systems evaluated independently.
Is Google-Extended the same as Googlebot?
No, and this is a common point of confusion. Googlebot indexes your site for Google Search. Google-Extended controls only whether your content can be used to train future Gemini models and for grounding in Gemini apps — per Google’s own documentation, it has no effect on Search inclusion or ranking either way. You can block Google-Extended with zero impact on your Google Search visibility.
Do AI crawlers slow down my site for real visitors?
They can, particularly on shared hosting with limited CPU allocation, if an aggressive or poorly-throttled crawler generates a traffic spike. This is a legitimate operational reason to rate-limit (via Crawl-delay) or block specific bots, independent of any content-policy decision. Monitoring your logs (Section 8) is the way to tell if this is actually happening on your site versus a theoretical concern.
How often does this list of bots change?
Frequently enough to check periodically — Anthropic restructured its single ClaudeBot into a three-bot framework in 2025, and new operators (or new bot names from existing operators) appear regularly as AI companies launch new products. Treat any AI bot directory, including this one, as a snapshot to verify against each operator’s current documentation before relying on it long-term.
Know Which Bot You’re Talking To.
Then Decide On Purpose.
The single biggest mistake in AI crawler configuration isn’t choosing wrong between blocking and allowing — it’s treating “AI bots” as one undifferentiated category and applying one blanket rule. Training crawlers, search/answer crawlers, and user-triggered fetchers do genuinely different things, and a policy that doesn’t distinguish between them usually gets at least one of the three wrong.
Start by looking at your actual logs, not assumptions. Decide your training-data policy separately from your AI-search-visibility policy. Implement through robots.txt first, escalate to server-level blocking only where compliance is unreliable, and keep your TTFB fast enough that the real-time, user-triggered fetches — the ones that happen while an actual person is waiting on an answer — have a chance to succeed at all.
Different bots.
Different decisions.