The Tech Billionaire Taking on OpenAI and Google | BBC News

Key Concepts

• The Internet Deal: The historical exchange where content creators publish online and, in return, receive either monetary compensation or fame/visibility.
• Answer Engines vs. Search Engines: The shift from search engines providing links (treasure maps) to AI-powered systems providing direct answers.
• AI Crawling Ratio: The metric indicating how many pages an AI system crawls for every visitor it sends to a website.
• Scarcity: The principle that a market cannot exist without a limited supply of a resource.
• Cloudflare's Decision: The policy to block AI bots from scraping data from websites protected by Cloudflare technology.
• Fair Exchange of Value: The ideal scenario where content creators are compensated for the use of their original content by AI systems.

The Changing Internet Deal and the Rise of Answer Engines

The internet's foundational deal has historically been the exchange of content for visibility or monetary gain. This model has driven the internet as we know it. However, this deal is undergoing a significant transformation, evident in the evolution of AI. Sci-fi visions of helpful robots providing direct answers, rather than links, are becoming reality with AI systems like ChatGPT, Anthropic, and Google's AI overviews. These are shifting from being "search engines" (treasure maps) to "answer engines."

The Impact on Media Companies and the Data Crawling Phenomenon

Matthew, a representative from Cloudflare, observed a growing concern among media companies. Initially, these companies, whose primary business is cybersecurity against state-sponsored attacks (e.g., from China, Russia, North Korea, Iran), reported a new threat: AI companies. Matthew's initial reaction was skepticism, viewing these as typical complaints about new technology from "luddites" or "journalists."

However, upon investigation, it was confirmed that AI companies were indeed impacting media businesses. The core mechanism involves how search engines and AI systems operate: they employ "crawlers" or "bots" to visit and copy content from the internet. This copied content is then used to build an index for search results or to train AI models.

The historical "deal" was that content creators allowed this crawling in exchange for traffic and visibility. For the past decade, Cloudflare has tracked the ratio of pages crawled by Google versus visitors sent back to publishers.

• 10 Years Ago: The ratio was approximately 2 pages crawled for every 1 visitor sent.
• Today: This ratio has significantly worsened, reaching nearly 20 pages crawled for every 1 visitor.

The primary driver of this change is the advent of AI overviews, which provide answers directly, eliminating the need for users to click through to the original content.

The situation is even more extreme with dedicated AI companies:

• OpenAI (ChatGPT): Reports a ratio of 1,500 crawls for every 1 visitor sent.
• Anthropic: Reports a staggering ratio of 40,000 crawls for every 1 visitor.

This imbalance represents a significant cost to content creators like the BBC and others, as they bear the expense of hosting and maintaining the crawled content without receiving commensurate traffic or monetization opportunities. As the internet increasingly moves towards answer engines, the existing deal between content creators and these platforms is unsustainable and requires re-evaluation.

Cloudflare's Decision: Creating Scarcity for Fair Exchange

In response to this evolving landscape and the plight of media companies, Cloudflare made a significant decision: to block AI bots from scraping data from websites protected by their technology.

This decision stems from the realization that for a sustainable market to emerge, scarcity is essential. Just as there's no market for air unless it's scarce (like underwater), there can be no market for content if it's freely available to all AI systems without compensation.

Cloudflare believes that by restricting AI bot access, they can:

1. Create Scarcity: Make content a more valuable and limited resource.
2. Foster a New Market: Encourage the development of a business model that rewards original content creation.
3. Protect Media Companies: Provide tools for media organizations, which may lack the technical sophistication of AI firms, to safeguard their data.

The goal is not to lock up media behind AI companies or to cause journalists to "starve," but to establish a "fair exchange of value" where original content creators are paid for their work. The first step in achieving this is to restrict the indiscriminate crawling by AI bots.

Addressing Criticisms and the Economic Argument

Cloudflare anticipates criticism that blocking AI innovation is detrimental. Arguments against this decision include:

• Liberal Argument: Users desire direct answers and the best possible AI services, as exemplified by wanting the best holiday recommendations from a chatbot.
• Economic Danger: Placing "blockers" on AI data access could hinder innovation and economic growth.

However, Cloudflare counters these points by highlighting that innovation in AI already involves costs and restrictions:

• Hardware Costs: Companies like Nvidia charge for essential components like chips. Jensen Huang, the CEO of Nvidia, requires payment for these resources.
• Talent Costs: Top AI researchers command salaries. Cloudflare argues against the notion of "slavery" where talent is not compensated.

Therefore, Cloudflare questions why AI companies should be exempt from paying for the "fuel" of their systems – the content created by others – when they readily pay for chips and researchers. The lack of compensation for content, while paying for other essential resources, is seen as illogical and unfair.