AI Bot & Crawler Traffic Statistics 2026
How much of the web is now automated, which AI crawlers generate the most requests, how many pages they take for every visitor they send back, and what that traffic costs the sites serving it. Every figure below carries a named source and a date, and live measurements are labeled with the exact window they were read.
Why “what percent of web traffic is bots” has three different answers
The three most-quoted sources on bot traffic disagree by nearly 14 percentage points. They are not contradicting each other — they are measuring different things. Here is the reconciliation, because picking the wrong figure is the most common error in reporting on this subject.
Original analysis by Paid Hosting · October 2026 · methodology · full source list
📋 What’s in this page
- Key Bot Traffic Statistics at a Glance
- How Much of Web Traffic Is Bots
- Which AI Crawlers Generate the Most Traffic
- Crawl-to-Refer Ratios
- What AI Crawlers Are Collecting
- How Site Owners Are Responding
- Bandwidth & Server Cost
- Referral Traffic & Publisher Impact
- Malicious Bots & Automated Attacks
- Sources & Methodology
Key Bot Traffic Statistics at a Glance
The headline numbers that define automated web traffic in 2026. Every figure links to a named report or a dated live measurement in the sources list.
How Much of Web Traffic Is Bots
This is the most-requested figure on the subject and the one most often misquoted, because the three authoritative sources report numbers between 39% and 53%. The difference is methodological, not a dispute about facts.
Cloudflare Radar’s live dashboard reported 39.2% of worldwide HTTP requests as bot traffic for October 19–25, 2026. Cloudflare’s own July 2026 report states that more than 50% of internet traffic is now non-human. Thales, analyzing full-year 2025 data across its customer base, puts bots at 53% of all internet traffic. Three reasons account for the gap. First, scope: Radar’s “bot” classification counts requests it identifies as automated, while “non-human traffic” is a broader category that includes AI agents acting on behalf of users, which Thales began counting as a separate third category in its 2026 report.Second, network: each vendor sees only its own customers’ traffic, and the mix of sites behind Cloudflare and Thales differs. Third, period: Radar reports a rolling seven-day window that moves daily, while the vendor reports are annual and lag by months. If you need one figure, say “roughly 40% to just over half of web traffic is automated, depending on how automation is defined” and cite the specific measurement you mean.
- Bots generated 39.2% of worldwide HTTP requests, with humans at 60.8%.Source: Cloudflare Radar, live worldwide reading for October 19–25, 2026
- More than 50% of traffic on the internet is now non-human — a threshold crossed for the first time this year.Source: Cloudflare, “Content Independence Day, one year on,” July 1, 2026
- Bots accounted for 53% of all internet traffic across 2025, with bad bots alone at 40% (up 3 percentage points year on year).Source: Thales 2026 Bad Bot Report, April 30, 2026 (13th edition, full-year 2025 data)
- Taking those two Thales figures together, roughly three-quarters of all bot traffic (75.5%) is classified as malicious rather than benign.Source: Paid Hosting calculation from Thales 2026 Bad Bot Report figures (40% ÷ 53%)
- The United States originates 44.2% of worldwide bot traffic — more than the next nineteen countries combined (40.9%). Germany follows at 5.6%, the Netherlands at 4.9%, Singapore at 3.8% and China at 3.0%.Source: Cloudflare Radar, October 19–25, 2026
- Google’s network AS396982 is the single largest source of bot traffic at 7.9%, followed by Amazon (AS16509) at 7.6%, a second Amazon network (AS14618) at 6.2%, Microsoft (AS8075) at 5.5% and Cloudflare (AS13335) at 5.4% — a reminder that most crawling originates from cloud infrastructure rather than residential networks.Source: Cloudflare Radar, October 19–25, 2026
- Generative AI reached 2.5 billion active users — over 30% of humanity — in three and a half years, adopted at more than twice the speed of smartphones.Source: Cloudflare, July 1, 2026
Which AI Crawlers Generate the Most Traffic
Five bots account for roughly three-fifths of all AI bot and crawler traffic. The ranking has shifted substantially in twelve months, which is why the older figures further down this section are worth reading alongside the current ones.
Share of AI bot and crawler traffic by bot
Top five most active AI bots as a share of AI bot and crawler HTTP requests, worldwide.
Source: Cloudflare Radar, worldwide, October 19–25, 2026.
Chart may be reproduced with attribution to paidhosting.com- The five most active AI bots — Googlebot, ClaudeBot, Meta-ExternalAgent, Applebot and Bingbot — together account for 60.7% of AI bot and crawler traffic. OpenAI’s GPTBot is not in the top five this week.Source: Paid Hosting calculation from Cloudflare Radar, October 19–25, 2026
- Within OpenAI’s traffic, GPTBot (the training crawler) generates 69.7% of requests, OAI-SearchBot 22.8% and ChatGPT-User 7.5%.Source: Cloudflare Radar, October 19–25, 2026
- Within Anthropic’s traffic, ClaudeBot generates 84.8%, Claude-SearchBot 14.9% and Claude-User 0.3%.Source: Cloudflare Radar, October 19–25, 2026
- Within Perplexity’s traffic, PerplexityBot generates 88.2% and Perplexity-User 11.8%.Source: Cloudflare Radar, October 19–25, 2026
- A year earlier the ranking looked very different: across April–July 2025, Meta’s AI crawlers alone generated 52% of AI crawler traffic on Fastly’s network, more than Google (23%) and OpenAI (20%) combined.Source: Fastly Q2 2025 Threat Insights Report, August 19, 2025
- In that same period, crawlers made up almost 80% of AI bot traffic and real-time fetchers the remaining 20% — and OpenAI’s bots generated 98% of all fetcher requests.Source: Fastly, August 19, 2025
- Fetcher traffic reached bursts of 39,000 requests per minute against individual sites — enough to degrade or overwhelm an unprepared origin server.Source: Fastly, August 19, 2025
- Crawler traffic is heavily concentrated in North America: nearly 90% of North American AI bot traffic came from crawlers, versus 72% in Latin America, 58% in Asia and 41% in Europe. Meta and Google sent over 90% of their crawler traffic to North American sites.Source: Fastly, August 19, 2025
Fastly’s 2025 report put Meta far ahead of every other crawler; Cloudflare’s 2026 live data puts Googlebot first, ClaudeBot second and Meta third. Both are accurate for what they measure. The two networks serve different customer bases in different regions, and Fastly’s figure covers AI crawlers specifically while Cloudflare’s covers AI bots and crawlers together, including search. The Fastly figures are also a year older. Use Cloudflare for a current reading and Fastly as a historical comparison point — not as competing claims about the same quantity.
Crawl-to-Refer Ratios
The crawl-to-refer ratio is the number of pages an operator’s bots request for every visitor it sends back to the site. It is the clearest single measure of the exchange between AI companies and the sites they read, and the spread between operators is enormous.
Pages crawled per visitor referred back
Ratio of HTML page crawl requests to HTML page referrals, by platform. Bars use a linear scale, so ratios below about 50:1 appear as thin slivers.
Source: Cloudflare Radar, worldwide, October 19–25, 2026. Mistral registered no referrals at all in this period.
Chart may be reproduced with attribution to paidhosting.com- Perplexity’s bots requested roughly 4,700 pages for every visitor referred back — the highest ratio of any major operator, ahead of OpenAI at roughly 1,800:1 and Anthropic at 510.6:1.Source: Cloudflare Radar, October 19–25, 2026
- Google’s ratio is 4.6:1, meaning Google returns a visitor roughly 1,000 times more often per page crawled than Perplexity does. DuckDuckGo (3.8:1) and Google are the only platforms below 5:1.Source: Paid Hosting calculation from Cloudflare Radar, October 19–25, 2026
- Mistral’s user agent registered no referrals whatsoever during the measurement week, making its ratio effectively infinite.Source: Cloudflare Radar, October 19–25, 2026
- Ratios are volatile week to week. Against the previous seven days, OpenAI rose 188.9%, ByteDance rose 38.3%, Perplexity rose 38.2% and Yandex rose 15.0%, while DuckDuckGo fell 17.4%. Any single reading should be quoted with its date.Source: Cloudflare Radar, October 19–25, 2026
- Around July 2025, Cloudflare observed crawl-to-referral ratios ranging from 118:1 to nearly 50,000:1 across leading AI companies — a far wider spread than the current range, indicating the extreme outliers have narrowed considerably.Source: Cloudflare, “Unmasking the crawls with Attribution Business Insights,” July 1, 2026
Crawl-to-refer ratios are among the most frequently cited statistics in coverage of AI and publishing, and they are also among the most volatile. The figures above moved by nearly 190% in a single week for one operator. They also measure HTML page requests against HTML page referrals on Cloudflare’s network only, so a site with a different traffic mix will see different ratios. Quote them with the operator, the source and the exact week, and treat any figure more than a few months old as historical rather than current.
What AI Crawlers Are Collecting
Cloudflare classifies crawler requests by declared purpose, which shows how much of the crawling is for model training rather than for search indexing that returns traffic to the source.
AI crawler traffic by declared purpose
Share of crawler request volume by the purpose the operator declares.
Source: Cloudflare Radar, worldwide, October 19–25, 2026.
Chart may be reproduced with attribution to paidhosting.com- 81.4% of AI crawler traffic involves model training, either as the sole declared purpose (42.7%) or combined with search (38.7%). Search-only crawling accounts for 12.0%.Source: Paid Hosting calculation from Cloudflare Radar, October 19–25, 2026
- 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025 (Radar’s October 19–25 window shows 42.7% training-only and 81.4% including mixed-use crawlers) — more than doubling in roughly a year.Source: Cloudflare, July 1, 2026
- More than one-third of crawler activity comes from mixed-use bots that do not distinguish between crawling for search and crawling for training, leaving site owners unable to allow one and refuse the other.Source: Cloudflare, July 1, 2026
- Content served to AI bots is overwhelmingly HTML at 75.3%, followed by plain text at 8.0%, JSON at 5.0% and images at 4.0%. Markdown accounts for 0.1% and documents for under 0.1%.Source: Cloudflare Radar, October 19–25, 2026
- Of the leading AI operators tracked by Cloudflare, Amazon, Anthropic, Meta and OpenAI both verify their bots and run distinct bots by purpose. Apple, Google and Microsoft verify but do not separate by purpose; ByteDance does neither.Source: Cloudflare Radar AI bot transparency tracker, read October 26, 2026
- Google’s combined discovery-and-AI crawler gives it access to roughly twice as much information as leading AI companies that separate the two functions.Source: Cloudflare, July 1, 2026
How Site Owners Are Responding
Two datasets show what site owners actually do about AI crawlers: what they declare in robots.txt, and what their servers return when a bot arrives. The two tell noticeably different stories.
Share of top-10,000-domain robots.txt files fully disallowing each bot
Calculated from 4,343 robots.txt files found across the top 10,000 domains. “Fully disallowed” excludes partial restrictions.
Source: Paid Hosting calculation from Cloudflare Radar robots.txt data as of October 19, 2026.
Chart may be reproduced with attribution to paidhosting.com- GPTBot is the most-blocked bot among the top 10,000 domains: 394 of 4,343 robots.txt files (9.1%) fully disallow it, and 588 (13.5%) restrict it fully or partially.Source: Paid Hosting calculation from Cloudflare Radar, data as of October 19, 2026
- Site owners block AI training crawlers roughly fifteen times more often than they block Google’s search crawler: 394 domains fully disallow GPTBot against 26 for Googlebot.Source: Paid Hosting calculation from Cloudflare Radar, data as of October 19, 2026
- Google-Extended, the opt-out token for Google’s AI training, is fully disallowed by 297 domains (6.8%) — while Googlebot itself is disallowed by 26. Site owners are drawing a clear line between being indexed and being used for training.Source: Paid Hosting calculation from Cloudflare Radar, data as of October 19, 2026
- Perplexity’s crawler is blocked far less often than the major training crawlers, at 4.2% fully disallowed against GPTBot’s 9.1%.Source: Paid Hosting calculation from Cloudflare Radar, data as of October 19, 2026
- In practice, servers refuse or throttle more traffic than robots.txt suggests: 18.9% of AI bot requests receive a 403 Forbidden and a further 3.2% receive a 429 Too Many Requests, for a combined 22.1% refused or rate-limited. Only 54.5% receive a normal 200 OK.Source: Paid Hosting calculation from Cloudflare Radar, October 19–25, 2026
- Cloudflare changed its default in 2025 so that AI training crawlers are blocked for all new domains unless the owner opts in — a network-level enforcement mechanism rather than the voluntary robots.txt standard.Source: Cloudflare, July 1, 2026
Only 9.1% of large sites fully disallow GPTBot in robots.txt, yet 22.1% of AI bot requests across Cloudflare’s network are refused or throttled at the server. The gap exists because robots.txt is an advisory standard that a crawler can ignore, while 403 and 429 responses are enforcement — and much of that enforcement now happens at the network layer through firewall rules and bot management products rather than through anything the site owner writes in a text file. Reporting that uses robots.txt adoption as a proxy for “how many sites block AI crawlers” will therefore undercount.
Bandwidth & Server Cost
Crawler traffic is a hosting cost before it is anything else. It consumes bandwidth, occupies server resources and is billed identically to human traffic — but the load it creates is disproportionate to its share of requests.
- Wikimedia’s bandwidth for serving multimedia content grew 50% between January 2024 and April 2025, driven principally by automated scraping rather than by growth in human readership.Source: Wikimedia Foundation, “How crawlers impact the operations of the Wikimedia projects,” April 1, 2025
- At least 65% of Wikimedia’s most resource-intensive traffic came from bots, while bots accounted for only about 35% of total pageviews — meaning automated traffic was roughly 1.9 times more expensive to serve than its request share implies.Source: Wikimedia Foundation, April 1, 2025; ratio calculated by Paid Hosting
- Serving markdown instead of HTML to AI bots reduces the median response to 13.2% of its original size — an 86.8% reduction in bytes transferred for the same content.Source: Cloudflare Radar, October 19–25, 2026; reduction calculated by Paid Hosting
- 75.3% of what AI bots receive is HTML, the heaviest of the common formats, and only 0.1% is markdown — so almost none of that potential saving is currently being realized.Source: Cloudflare Radar, October 19–25, 2026
- Bursts of AI fetcher traffic have reached 39,000 requests per minute against a single site, a rate comparable to a denial-of-service event for an unprepared origin server.Source: Fastly Q2 2025 Threat Insights Report, August 19, 2025
- Indiscriminate crawling creates unnecessary bandwidth burden for publishers and wastes compute for AI companies, according to Cloudflare, which is investing in freshness signals specifically to reduce redundant crawling.Source: Cloudflare, July 1, 2026
There is no reliable published figure for the average dollar cost of AI crawler traffic to a typical website, and any number presented as one should be treated with suspicion. What the evidence does establish is directional and well documented: crawler traffic consumes disproportionately more bandwidth than its share of requests, it concentrates on the most expensive resources to serve, and it arrives in bursts that size infrastructure requirements rather than averages. Site owners assessing the impact should measure their own logs rather than apply a general figure.
Referral Traffic & Publisher Impact
The counterpart to rising crawl volume is falling referral traffic. The exchange that funded the open web — access to content for visitors in return — is measurably weakening.
- Google users who encountered an AI summary clicked a traditional search result in 8% of visits, against 15% for users who did not see one — close to half the click rate.Source: Pew Research Center, July 22, 2025 (browsing data from 900 U.S. adults, March 2025)
- Users clicked a link inside the AI summary itself in just 1% of visits to pages containing one.Source: Pew Research Center, July 22, 2025
- 26% of visits to a search page with an AI summary ended the browsing session entirely, against 16% for pages with only traditional results.Source: Pew Research Center, July 22, 2025
- 58% of respondents encountered at least one AI-generated summary in their Google searches during March 2025.Source: Pew Research Center, July 22, 2025
- For every hour spent searching for information online, only about 15 minutes is now spent on the open web.Source: Cloudflare, July 1, 2026
- Some of the most heavily crawled site categories have seen human traffic fall by as much as 40% in less than a year.Source: Cloudflare, July 1, 2026
- Google still accounts for approximately 88% of referral traffic, making it simultaneously the largest source of visitors and a growing substitute for them.Source: Cloudflare, July 1, 2026
- More than 50 publisher–AI licensing agreements have been signed since 2023 as publishers seek to replace lost referral revenue.Source: Cloudflare, July 1, 2026
Malicious Bots & Automated Attacks
Not all automated traffic is crawling for content. The majority of bot traffic is classified as malicious, and AI has sharply accelerated the volume.
- Bad bots accounted for 40% of all internet traffic across 2025, up 3 percentage points year on year.Source: Thales 2026 Bad Bot Report, April 30, 2026
- Thales blocked 17.2 trillion bad bot requests during 2025 — an average of roughly 47 billion per day.Source: Thales 2026 Bad Bot Report, April 30, 2026; daily average calculated by Paid Hosting
- Daily AI-driven bot attacks rose more than twelvefold in a single year, from about 2 million per day in 2024 to 25 million per day in 2025 — though Thales notes its AI bot detection coverage also expanded during 2025.Source: Thales 2026 Bad Bot Report, April 30, 2026
- Financial services was the most targeted industry, absorbing 24% of all bot attacks.Source: Thales 2026 Bad Bot Report, April 30, 2026
- Retail — not financial services — is the sector most targeted by AI-driven bots specifically, at 20% of AI bot attacks; Thales notes that sites with dynamic pricing, limited inventory or high-demand promotions attract continuous automated activity.Source: Thales 2026 Bad Bot Report, April 30, 2026
- The most common API threats are data leakage (26%), remote code execution or remote file inclusion (13%), business logic abuse (13%), automated attacks (9%) and path traversal or local file inclusion (9%).Source: Thales 2026 Bad Bot Report, April 30, 2026
- Detectable AI traffic represents only a fraction of total AI-enabled activity, because attackers can deploy self-hosted models that never identify themselves — creating a permanent gap between measured and actual volume.Source: Thales 2026 Bad Bot Report, April 30, 2026
Cite this page
https://www.paidhosting.com/bot-traffic-statistics/
Figures, charts and calculations on this page may be reproduced with attribution and a link to this page. Where a statistic is attributed to a third party, please cite that source directly. Press and data inquiries: [email protected]
Sources & Methodology
Every statistic on this page has been read directly from the source listed below and carries its publisher and publication date. Live dashboard measurements are labeled with the exact window they cover and the date they were read, because those figures change continuously — a reader checking Cloudflare Radar today will see a different number than the one quoted here, and that is expected rather than an error. Figures marked as Paid Hosting calculations are arithmetic performed on the published source data, with the inputs stated so the working can be checked. Where sources disagree, we explain the methodological reason rather than choosing one and omitting the others. This page is updated as new data is published; live figures are re-read monthly. This page and our AI Agent Traffic analysis both read Cloudflare Radar, and each states the seven-day window it covers. Where the two pages cover different windows, figures for the same metric will differ. Both are accurate for the window stated beside them; neither supersedes the other.
- Cloudflare Radar — Bot Traffic, AI Insights and
robots.txtdatasets (radar.cloudflare.com). Live figures on this page were read on October 26, 2026 and cover October 19–25, 2026 worldwide, exceptrobots.txtdata, which Cloudflare last updated October 19, 2026. - Cloudflare — “Content Independence Day, one year on: building the business model for the agentic Internet,” July 1, 2026
- Cloudflare — “Unmasking the crawls with Attribution Business Insights,” July 1, 2026
- Thales — 2026 Bad Bot Report: Bad Bots in the Agentic Age, April 30, 2026 (13th edition; based on full-year 2025 bot activity across Thales and Imperva networks)
- Fastly — Q2 2025 Threat Insights Report, August 19, 2025 (AI bot traffic observed mid-April to mid-July 2025)
- Pew Research Center — “Google users are less likely to click on links when an AI summary appears in the results,” July 22, 2025 (browsing data from 900 U.S. adults who consented to share activity, March 2025)
- Wikimedia Foundation — “How crawlers impact the operations of the Wikimedia projects,” Diff, April 1, 2025
A note on vendor data: Cloudflare, Thales and Fastly each measure only the traffic crossing their own networks, and each also sells products that mitigate bot traffic. Their figures are the best available primary measurements of automated traffic at scale, and we cite them as such, but they are not neutral third-party audits and none of them observes the whole internet.