The Visitors That Act Instead of Read
Crawlers collect pages. Agents do things — fetch a page because a person asked a question thirty seconds ago, follow a link, complete a task. That distinction is now measurable, and the numbers show a web where humans are already a minority of page requests and roughly one in five AI requests is refused at the door.
On this page
Agents are not crawlers, and the data now separates them
Most reporting on AI traffic treats every non-human request as one category. It isn’t. A training crawler visits on its own schedule, in bulk, to build a dataset. An agent visits because a person is waiting for an answer, fetches what that specific task needs, and leaves. The infrastructure consequences are opposite: crawlers are a bandwidth problem you can throttle, agents are a latency problem you cannot.
The finding worth sitting with is a governance gap rather than a traffic number. Site owners who add an AI bot to robots.txt generally believe they have opted out. For crawlers they have. For agents it depends entirely on which operator is asking, because the two largest publish directly conflicting positions on whether robots.txt applies to user-initiated requests at all — and both positions are stated in their own public documentation.
Cite this analysis
https://www.paidhosting.com/ai-agent-traffic/
Findings and charts on this page may be reproduced with attribution and a link to this page. Press and data enquiries: [email protected]
Key figures at a glance
How much AI traffic is agent-driven
Cloudflare asks AI operators to declare why their bots are fetching a page, and publishes the breakdown. This is currently the only public measurement that isolates real-time, user-triggered requests from bulk collection. Training remains dominant at 40.2%, with a further 34.8% declared as combined training and search. User action — the agent slice — sits at 5.8%.
AI bot traffic by declared crawl purpose
Share of AI bot and crawler HTTP requests, worldwide, 4–10 August 2026.
Source: Cloudflare Radar AI Insights, worldwide, 4–10 August 2026.
Chart may be reproduced with attribution to paidhosting.comCrawl purpose is self-declared by the AI operator, not independently verified. 1.8% of traffic carries no declaration at all. The 5.8% figure should therefore be read as the share of AI bot traffic that operators describe as user-triggered, which is a floor rather than a precise measure — an operator with no incentive to distinguish the two may declare everything as training. It is nonetheless the best public measurement available, and no equivalent exists elsewhere.
Who is actually requesting your pages
Broken down by client type, HTML page requests are already majority non-human. Non-AI bots — monitoring, scrapers, security scanners, SEO tools — account for 49.9%. Humans account for 39.5%. AI bots account for 7%, with a further 3.6% classified as mixed purpose.
HTML page requests by client type
Share of HTML page requests, worldwide, 4–10 August 2026.
Source: Cloudflare Radar AI Insights, worldwide, 4–10 August 2026.
Chart may be reproduced with attribution to paidhosting.comCloudflare’s headline bot share, measured across all HTTP requests, is 35.7% for the same period. The 39.5% human figure above is not in conflict with it — it measures a different denominator. Counting every HTTP request includes images, scripts, stylesheets and API calls, which humans generate in large numbers per page view. Counting HTML page requests only asks who requested a page, and on that measure automated clients outnumber people. Both figures are correct; they answer different questions, and the difference between them is why AI traffic reporting so often appears contradictory.
Does robots.txt control agents? The operators disagree
Both OpenAI and Anthropic now run three separate bots along the same lines: one for model training, one for search indexing, and one for real-time user-initiated requests. The convergence on that three-way split is itself notable. What has not converged is whether the third category respects robots.txt.
OpenAI’s documentation states that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply
. Anthropic’s documentation takes the opposite position, stating that disabling Claude-User prevents its system from retrieving content in response to a user query — while noting this may reduce a site’s visibility in user-directed search.
| Operator | Training bot | Search bot | User-initiated agent | robots.txt applies to the agent? |
|---|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User | May not apply, per OpenAI documentation |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User | Applies — disabling prevents retrieval |
Source: OpenAI crawler documentation and Anthropic support documentation (dated 8 April 2026), both read 11 August 2026. Compiled by Paid Hosting.
- OpenAI operates three documented user agents — GPTBot for training, OAI-SearchBot for search, and ChatGPT-User for user-initiated actions — each controllable independently.Source: OpenAI crawler documentation, read 11 August 2026
- Anthropic operates the equivalent three — ClaudeBot, Claude-SearchBot and Claude-User — with the same independent-control model.Source: Anthropic support documentation, 8 April 2026
- OpenAI additionally notes that robots.txt changes can take around 24 hours to affect its search systems.Source: OpenAI crawler documentation, read 11 August 2026
- Both operators publish IP ranges for their bots, allowing verification independent of the user-agent string, which is trivially spoofable.Source: OpenAI and Anthropic documentation, read 11 August 2026
Every measurement of agent traffic depends on agents identifying themselves honestly, and every control depends on them honouring a file they are not obliged to read. A site owner who blocks GPTBot has opted out of training. They have not necessarily opted out of being fetched when a user asks a question — and whether they have depends on which company built the agent. This is a policy divergence, not a technical one, and it is documented by the operators themselves.
Agents mostly do not render JavaScript
Independent testing by Vercel and MERJ, published December 2024, found that none of the major AI crawlers rendered JavaScript — including OpenAI’s bots, Anthropic’s ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. The exceptions were Google’s Gemini, which uses Googlebot’s infrastructure, and AppleBot, which renders through a browser-based crawler.
The same analysis found AI crawlers fetching JavaScript files without executing them — 11.5% of ChatGPT’s fetches and 23.8% of Claude’s, on the sites studied. For a site whose content is assembled client-side, the practical consequence is that an agent may retrieve the page and see almost nothing.
- On the sites analysed, ChatGPT’s crawler fetches were 57.7% HTML, while Claude’s were 35.2% images and 23.8% JavaScript files.Source: Vercel and MERJ, “The rise of the AI crawler,” 17 December 2024
- Googlebot’s fetches were more evenly spread at 31.0% HTML, 29.3% JSON, 20.8% plain text and 15.3% JavaScript.Source: Vercel and MERJ, 17 December 2024
- At the time of that study, GPTBot generated 569 million requests and ClaudeBot 370 million across Vercel’s network in a single month, together around 20% of Googlebot’s 4.5 billion.Source: Vercel and MERJ, 17 December 2024
The Vercel and MERJ analysis was published in December 2024 and reflects crawler behaviour at that time. Rendering capability is exactly the kind of thing operators change without announcement, so this should be read as the most recent published independent test rather than a guaranteed description of current behaviour. We include it because no more recent independent rendering study has been published, and because the volume figures give a useful second network’s view alongside Cloudflare’s.
What happens when an agent arrives
Only 60.5% of requests from AI bots and crawlers receive a successful response. 16.3% are refused outright with a 403, and a further 3% are rate limited with a 429. Combined, roughly one in five AI requests is actively turned away rather than served.
HTTP response status codes returned to AI bots and crawlers
Worldwide, 4–10 August 2026.
Source: Cloudflare Radar AI Insights, worldwide, 4–10 August 2026. Remaining codes account for 3.8%.
Chart may be reproduced with attribution to paidhosting.com- 16.3% of responses to AI bots are HTTP 403 Forbidden, indicating an explicit block rather than a missing page.Source: Cloudflare Radar, 4–10 August 2026
- A further 3.0% receive HTTP 429, indicating rate limiting rather than outright refusal.Source: Cloudflare Radar, 4–10 August 2026
- Combined, 19.3% of AI bot requests are refused or throttled rather than served content.Source: Paid Hosting calculation from Cloudflare Radar data, 4–10 August 2026
- 7.5% receive a 404, and 7.8% receive a 301 or 302 redirect.Source: Cloudflare Radar, 4–10 August 2026
A 403 to a training crawler withholds your content from a dataset. A 403 to an agent withholds your content from a person who asked a question and is waiting for an answer. Because the two arrive over similar infrastructure and are often caught by the same rule, site owners frequently make the second decision without intending to. Cloudflare’s crawl purpose declarations exist precisely so the two can be separated, but acting on them requires configuring for it deliberately.
The payload problem
What gets returned to AI clients is overwhelmingly HTML built for browsers — navigation, styling, scripts, tracking — almost none of which an agent uses. 72.8% of responses are HTML. Markdown, which strips a page to the text an agent actually parses, accounts for less than 0.1%.
Content types returned to AI bots and crawlers
Grouped by MIME type category, worldwide, 4–10 August 2026.
Source: Cloudflare Radar AI Insights, worldwide, 4–10 August 2026.
Chart may be reproduced with attribution to paidhosting.com- Serving markdown instead of HTML to AI clients reduces the median response to 7.2% of its original size.Source: Cloudflare Radar, 4–10 August 2026
- That is a reduction of roughly 93% in bytes transferred, or approximately a fourteen-fold saving.Source: Paid Hosting calculation from Cloudflare Radar data, 4–10 August 2026
- Despite the saving being available and measured, markdown accounts for less than 0.1% of responses returned to AI bots and crawlers.Source: Cloudflare Radar, 4–10 August 2026
Which operators generate the traffic
AI bot traffic remains concentrated among a small number of operators. Googlebot leads at 20.9% of AI bot and crawler requests, followed by Meta-ExternalAgent at 13.7% and ClaudeBot at 12.7%.
Top five AI bots by share of AI bot traffic
Worldwide, 4–10 August 2026.
Source: Cloudflare Radar AI Insights, worldwide, 4–10 August 2026.
Chart may be reproduced with attribution to paidhosting.comWhat this means for hosting
Agent traffic behaves differently from both human and crawler traffic in ways that matter for capacity planning.
- It is latency-sensitive. A person is waiting on the other end of a user-action request, so a slow response degrades an experience rather than merely delaying a dataset.Inference from Cloudflare Radar crawl purpose definitions
- It generates no ad impressions and no analytics sessions. Agent requests consume origin resources without appearing in most measurement tools that hosting plans are sized against.Paid Hosting analysis
- It defeats caching more often than human traffic. Agents fetch specific pages relevant to a single task rather than following common browsing paths, so cache hit rates are lower.Paid Hosting analysis
- It is already a meaningful share of page requests. AI bots account for 7% of HTML page requests, and non-AI automated clients for 49.9%.Source: Cloudflare Radar, 4–10 August 2026
The latency, analytics and caching points above are our analysis of what the measured data implies, not figures published by Cloudflare or anyone else. We have labelled them as such rather than dressing them as statistics. No public dataset currently measures agent cache hit rates or the analytics gap directly — if one appears, this section will be updated to cite it.
Sources & Methodology
This page draws on three independent classes of source. Traffic measurements come from Cloudflare Radar AI Insights, worldwide, for the seven-day period 4–10 August 2026, retrieved 11 August 2026 — a large sample across Cloudflare’s network, but not a census of the internet. Independent rendering and content-type analysis comes from Vercel and MERJ’s December 2024 study, measured on a different network with a different site population. Bot taxonomy and robots.txt policy come first-hand from OpenAI’s and Anthropic’s own published documentation, read 11 August 2026.
Using more than one network matters here: Cloudflare’s sample skews toward the long tail of the web, Vercel’s toward modern JavaScript applications. Where they agree, the finding is more robust than either alone. Where our sources are the operators themselves, we have cited the documentation rather than paraphrasing it into a statistic.
Crawl purpose is self-declared by AI operators rather than independently verified. Figures for user action should be read as a floor. Percentages are of AI bot and crawler traffic unless stated otherwise; where a figure describes all HTML page requests or all HTTP requests, this is stated inline, because conflating those denominators is the single most common source of error in reporting on AI traffic.
Where we have combined published figures or drawn an inference rather than citing a measurement, this is attributed to Paid Hosting inline. Radar figures move week to week; this page is reviewed monthly and the date range above is updated each time. This page and our Bot Traffic Statistics analysis both read Cloudflare Radar, but over adjacent seven-day windows — 4–10 August here, 2–8 August there. Figures for the same metric therefore differ slightly between the two pages. Both are accurate for the window stated beside them; neither supersedes the other.
- Cloudflare Radar — AI Insights: HTTP traffic by bot, worldwide, 4–10 August 2026
- Cloudflare Radar — AI Insights: crawl purpose, worldwide, 4–10 August 2026
- Cloudflare Radar — AI Insights: HTML page requests by client type, 4–10 August 2026
- Cloudflare Radar — AI Insights: response status distribution, 4–10 August 2026
- Cloudflare Radar — AI Insights: content type distribution, 4–10 August 2026
- Cloudflare Radar — AI Insights: markdown for agents savings, 4–10 August 2026
- Cloudflare Radar — Bot traffic: bot versus human share, 4–10 August 2026
- Vercel and MERJ — “The rise of the AI crawler,” 17 December 2024
- OpenAI — Overview of OpenAI Crawlers, developer documentation, read 11 August 2026
- Anthropic — “Does Anthropic crawl data from the web…”, support documentation, 8 April 2026
Last reviewed 11 August 2026. Cloudflare Radar updates continuously; figures reflect the seven-day window stated above.