'Way bigger than COVID': The graph that explains why AI is going to be so huge

Key Concepts

• AI Benchmarks: Mathematical measures used to evaluate the performance of AI models through standardized tests.
• Exponential Trend: A pattern of growth where the rate of increase itself increases over time, often described as "nothing, nothing, nothing, then everything."
• Human Equivalent Time (50% Success Rate): A metric used to compare AI performance to human performance on specific tasks, indicating the time it takes for an AI to achieve a 50% success rate on a task compared to the average human time.
• Intellectual Tasks: Tasks that primarily involve cognitive abilities, such as problem-solving, reasoning, and creativity, as opposed to physical or social tasks.
• Turing Test: A test of a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human.

AI Progress and Benchmarks

The video highlights the rapid and significant advancements in Artificial Intelligence (AI), noting weekly releases from major AI companies and the impressive capabilities of these new models. This progress is not just anecdotal; it is supported by mathematical measures called benchmarks. These benchmarks function like exams for AI models, testing their performance before release. The results consistently show improving performance, indicating a steady upward trend in AI capabilities. The speaker emphasizes that the current state of AI is its "worst," with future improvements expected.

Expert Perspective: Mark Warner and Exponential Growth

To gain a deeper understanding of these advancements, the speaker interviews Mark Warner, a prominent AI founder in London and a scientist with extensive experience in the field since its early days. Warner explains the significance of AI benchmarks using a specific experiment from an organization called Meter.

Meter Experiment Details:

• Tasks: Approximately 170 software development tasks.
• Human Performance: 40-odd humans performed these tasks, and their average completion times were recorded.
• AI Performance: Various AI models from 2019 to the present were tested on these tasks. The metric used was the human equivalent time to achieve a 50% success rate.
• Findings: The experiment revealed a strong exponential trend in AI performance improvement over time. This means that AI capabilities are not just increasing linearly but are accelerating.

Warner draws a parallel between this exponential growth and his past experience modeling the spread of the coronavirus in March 2020. He observed that the virus followed an exponential curve ("nothing, nothing, nothing, then everything"), which led to critical interventions to prevent the collapse of the NHS. This historical context underscores his concern and warning about the current AI trajectory.

Reliability and Generalization of AI Performance

While acknowledging the intuitive feeling of massive progress in software engineering, Warner notes that the exact details of evaluation metrics and tasks can be more challenging to measure precisely. The field lacks a fully developed theory of intelligence, necessitating the creation of numerous small tests to piece together a broader understanding.

A key unknown is whether this exponential trend will continue and how it can be generalized across other types of tasks beyond software engineering. If LLMs (Large Language Models) remain confined to software development, the impact, while significant for that sector, might be limited for society at large. However, if this trend extends to a broad proportion of intellectual tasks, the implications become profoundly meaningful.

Warner observes that areas like human-like conversation and generating realistic images also appear to be following a similar direction. The challenge now is to create tests that AI cannot immediately saturate, indicating the need for increasingly sophisticated evaluation methods. He speculates that the Turing Test might have been surpassed by AI models around the time ChatGPT was released, and that AI has become as capable as human efforts for a significant number of tasks within a year or two.

Future Projections and Limitations

When asked about the future shape of the performance curve, Warner states that it cannot continue upwards indefinitely. A naive extension of the trend would require more energy than exists on the planet for training models. Therefore, the trend must eventually plateau. Faculty is actively researching when and where this plateau might occur.

However, he estimates that there are likely at least another five years of this exponential growth. This means that the tasks AI is capable of are doubling every seven months. Even with a five-year projection, this leads to a substantial increase in AI capabilities, potentially pushing the performance line "through the ceiling."

Real-World Application Caveats:

The speaker raises a crucial point: the 50% success rate is not sufficient for most real-world applications, which often require 99% or 99.9% success rates. This significantly shifts the timeline and slows down the practical implementation of AI for critical processes. Nevertheless, the need to prepare for these advancements is paramount, as unpreparedness for exponential trends can lead to severe consequences, as seen during the COVID-19 pandemic.

Societal and Economic Implications

Warner emphasizes that if this trend continues and generalizes, it will be the biggest technology trend of our lifetimes, potentially leading to the biggest economic, social, and cultural shifts. He clarifies that this trend primarily applies to intellectual tasks and not physical or social tasks.

Potential Upsides:

• Medicine: Enabling breakthroughs in drug discovery and personalized treatments.
• Education: Providing every child with an expert-level personal tutor at near-zero marginal cost.
• Healthcare: Offering individuals a "GP in their pocket" for instant symptom querying and diagnosis, potentially surpassing current human capabilities.

Potential Downsides and Disruptions:

• Value Shift: Changing what is considered valuable in society.
• Disruption: Causing significant changes and requiring adaptation.

Warner stresses the importance of serious consideration and planning to manage these changes, aiming to maximize the upsides and minimize the downsides. He compares the potential disruption to COVID-19, suggesting that if the AI trend continues and generalizes, it could be way bigger than COVID, representing a more permanent reshaping of how everything operates.

Technological Analogy

To put the rapid advancement into perspective, Warner likens the progress from basic AI capabilities to human-like voice interaction to going from the "first flight to something like Concord in a seven-year period." This highlights the immense speed and scale of development, with the potential for even more advancements to follow.

The speaker concludes by reiterating that Warner's focus was on the practical improvement of AI in performing human cognitive tasks, not on abstract concepts like AGI or consciousness. The key takeaways are the doubling of AI capabilities every seven months and the estimated five more years of this trend, posing significant questions about jobs, media, and our understanding of truth. While the answers are uncertain, the speaker agrees that these are critical issues that require serious thought and discussion.