AI has broken the internet’s economic bargain – here’s how we fix it

The internet – that vast library of human knowledge and creativity – was built on a simple, powerful exchange: share content and the network sends traffic in return. This fundamental bargain sustained the digital economy for decades, fueling journalism, culture, commerce and innovation.

Today, that bargain is broken, with the architecture of the web undergoing a fundamental rewiring and a different deal for content and commerce.

As chief strategy officer at Cloudflare, I have a front-row seat to the massive shifts occurring at the digital infrastructure layer. Our data reveals a dramatic escalation in the “bot wars,” a conflict that demands immediate attention from global leaders, policymakers and innovators.

The critical question facing us is this: how do we protect the high-quality, original content and commerce that fuels our digital economy?

Web crawlers have acted as digital librarians, indexing information for searchability, since the 1990s. However, the last two years has seen the emergence of artificial intelligence (AI) crawlers.

Unlike their predecessors, which drove traffic back to the source, these new AI bots absorb content to train and inform models that answer users directly.

On one hand, this has introduced an incredible user experience, providing instant, synthesised answers that remove the friction of traditional searching. On the other hand, these bots consume the web’s raw material without sending users to the original source, threatening the very ecosystems that feed them.

To maintain a vibrant and accessible digital economy, we must balance this seamless user experience with sustainable business models that fairly value and support the creators of original content.

In terms of scale, according to recent Cloudflare Radar data, combined traffic from search and AI crawlers grew by 19% in 2025. Some of the largest AI crawlers increased their request volume by more than 300% in a single year.

Even more staggering is what these bots returned in traffic to the website owners they crawl: almost nothing.

For example. In the first week of December 2025, for every 28,400 HTML pages requested by a prominent AI bot, it referred back just one visit to the original site. Both new and established models show a stark mismatch between what they consume and what they contribute to the content creator.

Consider the asymmetry: A single human visit adds value to a website e.g. through advertising impressions, product purchases, subscription etc; a single bot visit that replaces a billion human visits threatens its existence.

The original agreement with creators was: give us your content, and in exchange, we'll send you traffic – the currency of the digital economy.

Now, internet users are transitioning from search engines that provide a list of links for users to manually explore to answer engines powered by AI. This fundamental shift means users now expect and receive direct, synthesized answers to complex questions, accelerating the trend of "zero-click" searches.

When traffic referral drops by a factor of 750 or even 30,000 times compared to the "old Google" model, the business model collapses. The war for the internet's future is happening now and it's being fought by bots.

This isn't about opposing the development of AI, which still offers the potential for a transformative digital renaissance and democratization. To deliver on that promise, AI needs data, context and training to be useful but it also needs a healthy ecosystem where original content is not treated as an infinite, free resource.

If high-quality content disappears because creators cannot sustain their work, AI models will eventually feed on noise, duplication and low-value material, leading to a less reliable and ultimately less useful internet for all.

Unconsented crawling is not sustainable. The internet can only remain an open and diverse space if we establish new rules and the underlying infrastructure that delivers control and choice to creators, merchants and businesses.

This shift involves three crucial pillars:

To establish a functioning market, we must introduce clarity about who is accessing a site and why. AI platforms must clearly disclose the nature of their crawling. Is the bot gathering information for general search or is it exclusively hoarding data for a proprietary training model?

Site owners need this information to make informed decisions about granting permission. Without this fundamental transparency, AI companies aren’t just collecting data; they are undermining the trust that holds the web together.

Once a retailer or creator knows the identity and intent of the visiting bot, they must have the power to define the terms of engagement. Whether you're a publisher or a retailer, you must be able to block, allow or charge for specific actions.

For example, a small business may want to charge for the right to use their product specifications to train a proprietary mode but want to allow indexing to answer a shopper’s questions as they’re researching products.

A new, sustainable equilibrium relies on enabling websites to control access and decide whether to charge for the value their content provides. This creates the possibility of pay-per-crawl models that compensate journalists, artists, and merchants for the fuel that powers these powerful AI engines.

In the past, search engines sustained the content ecosystem but this responsibility will increasingly fall to AI companies. To build a healthy digital economy, the internet's underlying infrastructure must evolve through more efficient models that deliver high-quality data and reciprocity, so creators, publishers and small businesses fuelling the digital world can thrive.

Are they able to surface the right content to models so their brand is represented accurately? Can retailers make product discovery even easier for consumers increasingly turning to AI for answers? Are there ways to prevent fraud when AI agents make purchases for humans? These are the questions we must ask ourselves to build an ecosystem that works for everyone.

This is not a stance against innovation; it’s a commitment to building a sustainable and fair digital economy. The future we must strive for is one where there are many AI companies, voices and sellers.

As leaders in the digital ecosystem, we have a responsibility to establish rules that encourage collaboration, innovation and mutual respect. AI can absolutely coexist with creators, merchants and small businesses of all types but only if the relationship is fair, consensual and transparent.