The History of AI Explained: Crash Course Futures of AI #1

Key Concepts

• Moore's Law: The observation that the number of transistors on a microchip doubles approximately every two years, leading to exponential growth in computing power.
• Artificial Intelligence (AI): Computer systems designed to emulate intelligent behavior.
• Narrow AI: AI systems designed to perform a single, specific task (e.g., playing chess, facial recognition).
• General Purpose AI: AI systems capable of performing a wide range of tasks and adapting to different goals.
• Symbolic AI: AI that relies on logic coded directly into its hardware, using hardcoded search and evaluation without actual learning.
• Neural Networks: Computer architectures inspired by the human brain, processing information through layers of connected nodes.
• Deep Learning: A subset of machine learning that uses neural networks to learn from large amounts of data.
• Transformer: A tool that enables certain AI models to process entire sequences of data simultaneously, improving efficiency.
• AI Benchmarks: Standardized tests used to measure and compare the performance of different AI models.
• Scaling Laws: Formulas that describe how an AI's performance changes with increases in data, compute power, or both.

The Exponential Growth of Computing and the Rise of AI

For half a century, computing power experienced exponential growth, transforming computers from refrigerator-sized machines in the 1960s to pocket-sized personal computers by the 2010s. This rapid advancement, predicted by Gordon Moore in 1965 as Moore's Law (doubling transistors every two years), enabled capabilities previously unimaginable. Today, the frontier has shifted to Artificial Intelligence (AI), a type of advanced computer system designed to emulate intelligent behavior. AI is a broad term encompassing both futuristic concepts and existing technologies like smartphone facial recognition.

Evolution of AI: From Narrow to General Purpose

AI can perform tasks similar to humans, such as learning from errors and making predictions, but also tasks beyond human capacity, like processing vast amounts of data instantly or generating hyperrealistic fake videos. The journey to current AI capabilities began in the 1950s.

Early AI: The Bernstein Chess Program and Symbolic AI

• Bernstein Chess Program (1957): Developed by IBM, this was an example of narrow AI, excelling only at playing chess. It used algorithms mimicking human strategies, evaluating the board, and simulating moves. While it could beat an inexperienced human, it took eight minutes per move.
• Symbolic AI: This approach, exemplified by early chess bots like the Bernstein program and IBM's Deep Blue (1997), relies on hardcoded logic and search algorithms. Deep Blue could evaluate 200 million chess positions per second and famously beat Grandmaster Garry Kasparov in 1997. However, symbolic AI systems did not truly learn or improve; they followed programmed instructions.

The Deep Learning Revolution: Neural Networks and Stockfish

The paradigm shifted with the introduction of neural networks, computer architectures that function similarly to the human brain, processing information through interconnected nodes. Unlike symbolic AI, neural networks can learn and adapt from experience.

• Stockfish (2020): This chess bot incorporated an efficiently updatable neural network (NNUE), fundamentally changing its performance.
• Deep Learning: This process relies on three key components:
  1. Data: The amount of information available for AI to learn from.
  2. Algorithm: The instructions guiding the learning process.
  3. Compute: Processing power, memory, and storage.
• NNUE Advantage: Stockfish's NNUE allows for incremental updates, freeing up processing space and enabling more sophisticated move evaluation. Since 2020, Stockfish has dominated computer chess championships.

The Era of General Purpose AI and Scaling Laws

The deep learning revolution of the 2010s laid the groundwork for general purpose AI, systems capable of diverse tasks like writing, image generation, summarization, and autonomous driving. This rapid advancement is facilitated by transformers, which allow AI to process data sequences holistically rather than sequentially.

Measuring AI Progress: Benchmarks and Saturation

To track AI's rapid progress, AI benchmarks are used. For narrow AI, these are task-specific (e.g., FIDE ratings for chess engines). For general purpose AI, benchmarks evaluate performance across multiple domains, such as answering academic questions, summarizing text, or writing creative content. When AI models consistently perform well on a benchmark, it's considered saturated. The speed at which benchmark saturation is occurring is described as "freaky fast."

Scaling Laws: Predicting Future AI Capabilities

Observing AI performance across various benchmarks reveals patterns, notably that AI neural networks with more data and compute generally perform better. This observation has led to the development of scaling laws, formulas that predict how AI performance will evolve with increased data and compute. These laws suggest that "bigger is better" in the realm of neural networks and deep learning, where pattern recognition from vast datasets surpasses human-coded instructions.

• Implications of Scaling Laws: These laws help explain why large tech companies like Microsoft and Google, with access to extensive data and processing power, develop the highest-performing generalized AI models. Scaling laws offer insights into AI's potential to transform economies or, conversely, lead to authoritarianism or the creation of endless Dave Matthews Band covers.

The Future of AI: Potential and Uncertainty

While scaling laws provide a framework for predicting AI's future capabilities, their longevity is not guaranteed due to potential bottlenecks. However, as algorithms improve and data and compute continue to increase, AI could achieve advanced feats like discovering new medical treatments or reaching superintelligence. The trajectory of AI development, as predicted by scaling laws, mirrors the exponential growth seen with Moore's Law, but the ultimate outcome remains uncertain, with the potential for both immense benefit and significant risk. The next installment will explore the societal implications of AI.