Bot traffic passed human traffic online for the first time in internet history this month, with Cloudflare’s network showing automated agents accounting for 57.4% of HTTP requests compared to 42.6% from people. Cloudflare co-founder and CEO Matthew Prince flagged the crossover on X on June 3, noting it arrived “faster than I predicted” (he had initially forecast the flip sometime in 2027).
The internet’s advertising model runs on one assumption: a human is on the other side of the browser window. For the majority of HTTP requests hitting servers, that assumption is now wrong. The revenue gap that creates has no clear industry answer yet.
57 Percent and Climbing
Cloudflare sits in front of roughly one-fifth of all websites globally, giving its Radar traffic tool one of the more reliable windows into what’s actually hitting servers across the web. The figures it reports count HTTP requests, the discrete calls a browser or agent fires each time it loads a page or retrieves data. That metric is where bots now win by a wide margin.
The exact crossover date is uncertain. Prince, responding to comments under his X post, wrote that the data is “a bit messy” and pinning the precise moment is difficult. His summary: “clearly on the other side now.” The Radar dashboard shows the bot share settled above 57% and still climbing.
On one dimension, humans still prevail. People spend more total hours on the internet, counted inside apps, video platforms, and social feeds. Those environments generate fewer discrete page requests than autonomous agents firing thousands of queries to complete a single task. Web analytics platforms, ad servers, and the billing systems behind content delivery networks all track HTTP request volume, not hours-of-use.
The bot traffic now dominant on the web isn’t simply the automated programs that have always existed. Before the generative AI era, automated traffic was mostly search engine crawlers, monitoring tools, and a tail of scrapers, a baseline Prince put at around 20% of web traffic at his SXSW appearance. The new class is agentic AI: autonomous programs that go beyond indexing pages to completing tasks on behalf of users, research, shopping, scheduling, writing. Each task may involve hundreds or thousands of page requests, far more than a search crawler generates indexing the same content once.
At the SXSW conference in Austin in March 2026, he told TechCrunch that bot traffic would exceed human traffic by 2027, citing “the rise of generative AI and its just insatiable need for data.” The actual crossover beat that estimate by more than a year.
From 20 Percent to Majority in Three Years
The generative AI era transformed automated traffic from a fixed minority into the web’s dominant force in roughly three years. HUMAN Security, a cybersecurity firm that processes more than one quadrillion digital interactions annually through its defense platform, published its 2026 State of AI Traffic report in March 2026. Automated traffic grew 23.5% year over year in 2025; human traffic grew 3.1%, an eight-to-one growth ratio favoring machines.
The mechanism is amplification. As Cloudflare’s CEO laid out at SXSW: a person shopping for a camera visits roughly five websites; the AI agent doing that task on their behalf might visit 5,000. One user intent, one thousand times the server requests. That asymmetry explains why the request-volume measure crossed over while hours-of-use did not.
The composition of that automated traffic also shifted. Training crawlers, the bots indexing content for AI model construction, made up 67.5% of AI-driven traffic by volume in 2025, per HUMAN Security’s data. Their share fell sharply as the year progressed while AI scrapers, programs retrieving content in real time to answer user queries, grew 597%. Content is being consumed in the moment a query runs.
- 7,851% year-over-year growth in traffic from AI agents and agentic browsers in 2025 (HUMAN Security 2026 report)
- 187% growth in monthly AI-driven traffic volumes from January to December 2025
- 95%+ share of AI-driven traffic in 2025 concentrated in retail and e-commerce, streaming and media, and travel and hospitality (HUMAN Security)
The Ad Model’s Broken Assumption
Why Bots Break the Loop
The internet’s business model is a short loop. Create content, attract visitors, sell those visitors’ attention to advertisers. Search engines, social networks, and most of what’s free to read run on a version of this logic. A human visiting a page has a chance of seeing the ad, maybe clicking it. A generation of media economics was built on that chain.
An AI agent scraping the same article to synthesize a user’s answer doesn’t see the ad. The output goes straight to the user; the source rarely gets a referral visit.
The business model of the internet was… create content, drive traffic, and then sell things… That was the business model. That breaks down because… bots don’t click on ads. Customers are trusting the output from the helpful robot. They’re not clicking through the footnotes.
Prince said it at the SXSW conference in Austin in March 2026, three months before his own network confirmed the crossover.
Publishers Measure the Damage
Publishers in the Digital Content Next network reported a median 10% year-over-year decline in Google Search referrals in 2025, with some outlets down as much as 25%. The CEO told NBC News in June that “the web actually shrank” between 2015 and 2025; a Pew Research Center study found 38% of webpages that existed in 2013 were no longer accessible a decade later, deleted or link-rotted.
On the cost side, AI crawlers impose direct infrastructure expenses on site operators, sending high volumes of page requests that consume bandwidth the publisher pays for. Cloudflare’s technical documentation notes that AI training crawlers source content from across the web while sending far fewer human visitors back to the sites they indexed.
“I think a lot of people kind of have said, ‘Well, this has proved sort of the dead internet theory.’ I think that’s actually kind of wrong at a lot, a lot of levels,” Cloudflare’s CEO told NBC News after the crossover announcement. “It’s given access to content creation to a much broader audience,” he added, pointing to AI’s role in lowering the cost of publishing.
The Crawl-to-Click Chasm
The math of how AI crawlers interact with the web explains why the ad model problem persists even as bot traffic climbs. A search engine crawler visits pages and sends humans back through search results. OpenAI and Anthropic’s systems work differently, pulling content to build or refresh AI responses without routing users to the source.
Cloudflare’s Radar platform tracked how many pages each major AI crawler visits for every single human visitor it refers back. Anthropic’s figure draws from Cloudflare’s crawl-to-click analysis published in October 2025; OpenAI and Google figures come from Cloudflare’s pay-per-crawl launch data released in July 2025:
| AI Crawler | Pages Crawled per Referred Human Visit | Data Period |
|---|---|---|
| 14 | 2025 | |
| OpenAI | 1,700 | July 2025 |
| Anthropic | 38,000 | July 2025 (after 87% decline from peak) |
Google’s crawler still operates closer to traditional search logic, indexing pages and sending humans back via results. Anthropic’s ratio fell 87% from an earlier peak after the company adjusted its crawling behavior under publisher pressure, landing at 38,000 pages per referred visit, the highest imbalance among major AI players at the time of Cloudflare’s measurement.
More than 95% of AI-driven traffic in 2025 concentrated in retail and e-commerce, streaming and media, and travel and hospitality, per HUMAN Security’s report.
Charging Machines to Read
The infrastructure-level response moved faster than most publishers expected. Cloudflare launched a pay-per-crawl marketplace in July 2025, giving website owners the ability to block AI crawlers, allow them for free, or charge a micropayment per visit. New accounts on Cloudflare’s platform now default to blocking AI bots. In April 2026, Cloudflare partnered with GoDaddy to integrate its AI Crawl Control tool into GoDaddy’s hosting platform, extending access controls across a large share of the web. Cloudflare chief strategy officer Stephanie Cohen described the goal as providing “essential underpinnings for a new Internet business model.”
The Toll-Booth Startups
A cluster of startups has built parallel payment systems for the same problem:
- TollBit charges AI companies a per-access transaction fee and passes 100% of that revenue to publishers, collecting a separate fee from the AI company side only
- Prorata.ai compensates rights holders in proportion to how much of their content appears in AI-generated answers, tying payment to influence rather than crawl count
- Microsoft’s Publisher Content Marketplace (PCM), launched in February 2026, lets publishers sell rights-cleared content at set prices to Microsoft and other AI developers on a pay-per-use model
Sam Altman, OpenAI’s chief executive, backed the micropayment model publicly in May 2026, describing agent-to-publisher payments as his answer for how the open web sustains itself. The position marks a shift from the lump-sum content licensing deals OpenAI has favored since ChatGPT’s launch. Condé Nast, Time, the Associated Press, and The Atlantic have signed up for Cloudflare’s pay-per-crawl system, per TechCrunch reporting.
The Governance Gap
The structural critics aren’t optimistic. A May 2026 report from the Open Markets Institute, analyzed by the Nieman Journalism Lab, argued that publishers face a “double bind”: the same companies stripping referral traffic are building the payment marketplaces designed to replace it. Researchers Courtney Radsch and Karina Montoya wrote that Big Tech is “occupying both sides of the value chain simultaneously” and called for regulatory scrutiny of platform operators setting de facto pricing standards in a market with no independent governance yet.
Nikhil Kolar, VP of publisher product at Microsoft AI, told the AdExchanger Programmatic AI conference in May 2026 that four out of five websites currently block AI bot traffic. Microsoft’s advice to those publishers: open the robots.txt files and treat AI crawlers as a traffic channel rather than a threat.
Cloudflare’s pay-per-crawl marketplace processed its first live transactions in the summer of 2025; both AI companies and publishers need to opt in for payments to flow, and most of the web has done neither.
