Andrej Karpathy: Software Is Changing (Again)

Software in the Era of AI

Key Concepts: Software 1.0, Software 2.0, Software 3.0, LLMs (Large Language Models), Partial Autonomy Apps, Generation-Verification Loop, Vibe Coding, Autonomy Slider, Context Management, Operating System Analogy, Technology Diffusion, Human-in-the-Loop.

Software Evolution: 1.0, 2.0, and 3.0

• Software 1.0: Traditional computer code written in languages like C++. Instructions for the computer to carry out tasks.
• Software 2.0: Neural networks, specifically the weights of a neural network. Instead of writing code directly, developers tune datasets and use optimizers to create the parameters. Hugging Face is presented as the GitHub equivalent for Software 2.0.
• Software 3.0: Large Language Models (LLMs) programmed via prompts, effectively using English as a programming language. This is a new kind of computer where prompts program the LLM.
  • Example: Sentiment classification can be done via Python (1.0), training a neural net (2.0), or prompting an LLM (3.0).

LLMs as Operating Systems

• LLMs are not just utilities like electricity but complex software ecosystems analogous to operating systems.
• Closed source providers (OpenAI, Gemini, Anthropic) are like Windows or macOS, while open source alternatives like the Llama ecosystem are akin to Linux.
• LLM is the CPU, context windows are the memory, and the LLM orchestrates memory and compute for problem-solving.
• LLM apps (like Cursor) can run on different LLM "operating systems" (GPT, Cloud, Gemini) similarly to how VS Code runs on Windows, Linux, or macOS.
• LLM compute is currently expensive, leading to centralized cloud-based time-sharing, reminiscent of 1960s computing. Personal computing revolution hasn't happened yet.
• Interacting with an LLM via text feels like using a terminal to access an operating system. A general GUI for LLMs hasn't been invented yet.

Unique Properties of LLMs: Reversed Technology Diffusion

• LLMs have flipped the direction of technology diffusion. Traditionally, governments and corporations are the first users of new technologies (electricity, cryptography, computing, flight, internet, GPS), with consumer adoption following later.
• With LLMs, consumer use (e.g., using LLMs to "boil an egg") is widespread, while corporations and governments are lagging in adoption.

LLM Psychology: People Spirits with Cognitive Deficits

• LLMs are "stochastic simulations of people" trained on vast amounts of text.
• They possess encyclopedic knowledge and memory, similar to the character in the movie Rainman.
• Cognitive deficits include:
  • Hallucinations (making up facts).
  • Insufficient self-knowledge.
  • Jagged intelligence (superhuman in some areas, making basic mistakes in others).
  • Entrograde amnesia (lack of learning and expertise consolidation over time). Context windows are like working memory that needs to be directly programmed.
• Security limitations: gullibility, susceptibility to prompt injection, potential data leaks.

Opportunities: Partial Autonomy Apps

• Instead of directly interacting with LLMs, dedicated apps are more efficient.
• Example: Cursor (AI-assisted coding) and Perplexity (AI-assisted search).
• Key features of successful LLM apps:
  • Context management handled by LLMs.
  • Orchestration of multiple LLM calls (embedding models, chat models, diff models).
  • Application-specific GUI for auditing and faster interaction.
  • "Autonomy slider" to adjust the level of AI control.

The Generation-Verification Loop

• Cooperation between AI (generation) and humans (verification) needs to be optimized for speed.
• Speeding up verification:
  • GUIs are crucial for visual auditing.
• Keeping the AI on the leash:
  • Avoid overly large diffs or overreactive agents.
  • Be concrete in prompts to increase the probability of successful verification.

Education with AI

• AI can be used in education, but it's important to keep it on a leash.
• Separate apps for teachers (course creation) and students (course consumption) with an auditable course artifact.

Analogies: Autopilot and Iron Man

• Tesla Autopilot as an example of partial autonomy with a GUI and an autonomy slider.
• The Iron Man suit as a metaphor: build "Iron Man suits" (augmentations) rather than "Iron Man robots" (fully autonomous agents). Focus on partial autonomy products with custom GUIs and UI/UX.

The Democratization of Programming: Vibe Coding

• LLMs programmed in English make everyone a potential programmer.
• "Vibe coding" is building something custom and winging it.
• Example: Kids vibe coding, which is seen as a gateway to software development.

Conclusion

The era of AI is fundamentally changing software development. Understanding the nuances of Software 1.0, 2.0, and 3.0, the operating system-like nature of LLMs, their psychological quirks, and the importance of human-in-the-loop systems are crucial for navigating this new landscape. The focus should be on building partial autonomy products that augment human capabilities and democratize programming through natural language interfaces.