Anthropic co-founder: AGI predictions, leaving OpenAI, what keeps him up at night | Ben Mann

Key Concepts:

• Superintelligence: AI exceeding human intelligence.
• AI Alignment: Ensuring AI goals align with human values.
• X-risk (Existential Risk): Risk of events that could destroy humanity.
• Economic Turing Test: Determining if an AI can perform a job as well as a human.
• Transformative AI: AI that causes significant societal and economic change.
• Scaling Laws: The relationship between model size, compute, and performance.
• Constitutional AI: Training AI using a set of principles or "constitution."
• RLAIF (Reinforcement Learning from AI Feedback): Training AI using feedback from other AI models.
• ASL (AI Safety Levels): A framework for assessing the risk associated with different levels of AI capability.
• Sycophancy: AI tendency to agree with or flatter users.

1. AI Talent War and Anthropic's Mission-Driven Approach:

• Meta is aggressively recruiting AI researchers with offers like $100 million signing bonuses.
• Anthropic is less affected because its employees are mission-oriented, prioritizing "affecting the future of humanity" over purely financial gains.
• Ben believes that for individuals who are driven by a desire to impact the future of humanity, Anthropic offers a more compelling opportunity than companies focused primarily on profit.

2. Scaling Laws and the Pace of AI Progress:

• The narrative that AI progress is slowing down is false; progress is accelerating.
• Model releases are becoming more frequent (every month or three months).
• Scaling laws continue to hold true, even across 15 orders of magnitude.
• The perception of slowing progress may be due to "time compression" and saturation of intelligence needed for simple tasks.
• The real constraint is developing better benchmarks to reveal the true extent of AI intelligence.

3. Defining AGI and the Economic Turing Test:

• AGI is a loaded term; "transformative AI" is preferred, focusing on societal and economic impact.
• The Economic Turing Test: If an AI can be hired for a job and perform it as well as a human, it has passed the test for that role.
• Transformative AI is achieved when AI passes the Economic Turing Test for 50% of money-weighted jobs.
• Passing this threshold would lead to massive GDP increases and societal change.

4. The Impact of AI on Jobs and the Future of Work:

• AI will cause a combination of skill-based and job-elimination unemployment.
• In a future with safe, aligned superintelligence, capitalism may look very different.
• A world of abundance where labor is almost free raises questions about the nature of jobs.
• The transition period from today's job market to a future of abundance is a concern.
• Customer service (Fin, Intercom) and software engineering (Claude Code) are already seeing significant AI impact.
• Customer service resolution rates are increasing, and software engineering teams are becoming more efficient.

5. Advice for Future-Proofing Careers:

• Be ambitious in using AI tools and willing to learn new ones.
• Don't use new tools as if they were old tools.
• When using Claude Code, ask for ambitious changes and try multiple times if it doesn't work initially.
• Even if it feels scary, take the risk and try new tools.
• Focus on skills that complement AI, such as creativity, curiosity, and kindness.

6. Anthropic's Founding and Prioritizing AI Safety:

• Ben and eight others left OpenAI in 2020 to start Anthropic.
• They felt that safety wasn't the top priority at OpenAI.
• Anthropic prioritizes safety above everything else, focusing on fundamental research and frontier development.
• Anthropic aims to be at the forefront of AI while ensuring safety.

7. The Tension Between Safety and Progress:

• Initially, safety and progress were seen as mutually exclusive, but they are actually convex.
• Working on alignment research has improved the character and personality of Claude.
• Constitutional AI allows for a more principled stance on AI values.
• The personality of Claude is directly connected to Anthropic's focus on safety.
• AI should understand what people want, not just what they say.

8. Constitutional AI Explained:

• The model generates an output, then evaluates whether it complies with constitutional principles.
• If the response violates a principle, the model critiques itself and rewrites the response.
• The model learns to produce the correct response out the gate.
• This process uses the model to improve itself recursively and align with desired values.
• The constitution is published to encourage a society-wide conversation about AI values.

9. The Importance of AI Safety and Alignment:

• Ben was inspired by science fiction and Nick Bostrom's "Superintelligence."
• Language models understand human values in a core way, making alignment more hopeful.
• The challenge is no longer just keeping "God in a box" but managing how people interact with powerful AI.
• Anthropic uses AI Safety Levels (ASL) to assess the risk associated with different levels of AI capability.
• ASL-3 poses a slight risk of harm, while ASL-4 could lead to significant loss of life, and ASL-5 could be extinction-level.
• Anthropic shares examples of its models doing bad things to raise awareness of potential risks.

10. Addressing Criticisms and the Doomer Perspective:

• Anthropic publishes research to make other labs aware of the risks.
• The company could be doing more attention-grabbing things if it didn't care about safety.
• Anthropic held back a consumer application for computer use due to safety concerns.
• Ben believes things are overwhelmingly likely to go well, but the downside risk is very large.
• It is crucial to work on alignment ahead of time, as it may be too late once superintelligence is achieved.
• Even a small chance of things going wrong is unacceptable when the future of humanity is at stake.

11. The Timeline for Superintelligence:

• Ben defers to superforecasters, citing the AI 2027 report (now forecasting 2028).
• A 50th percentile chance of hitting some kind of superintelligence in a small handful of years is reasonable.
• This forecast is based on the science of intelligence improvement, low-hanging fruit on model training, and scale-ups of data centers.
• The effects of superintelligence will take time to be felt throughout society.

12. Defining Superintelligence and Measuring Its Impact:

• Superintelligence can be defined by the Economic Turing Test or a significant increase in world GDP growth (above 10% per year).
• A 3x increase in GDP growth would be game-changing.

13. The Odds of Aligning AI Correctly:

• It's a really hard question with wide error bars.
• Anthropic's "Theory of Change" describes three worlds: pessimistic (alignment is impossible), optimistic (alignment is easy), and a world in between where actions are pivotal.
• Evidence suggests that we are in the middle world, where alignment research matters.
• Ben's best estimate for the chance of an X-risk or extremely bad outcome from AI is between 0 and 10%.
• It is extremely important to work on AI safety, even if the world is likely to be a good one.

14. RLAIF (Reinforcement Learning from AI Feedback):

• RLAIF involves training AI using feedback from other AI models, rather than humans.
• Constitutional AI is an example of RLAIF.
• RLAIF is more scalable than RLHF (Reinforcement Learning with Human Feedback).
• The challenge is to ensure that recursive self-improvement remains aligned.
• Drawing parallels to how humans and human organizations (corporations, science) recursively self-improve.
• Empiricism is key to enabling models to improve themselves.

15. Bottlenecks to Model Intelligence Improvement:

• The stupid answer is data centers and power chips.
• More compute, better algorithms, and more data are needed.
• The efficiency with which models run on chips also matters.
• Innovations in algorithms, data, and efficiency have led to a 10x decrease in cost for a given amount of intelligence.

16. Ben's Personal Perspective and the Impact of AI Safety Work:

• Inspired by Nate Soares' "Replacing Guilt," which discusses techniques for working through weighty topics.
• Embraces "resting in motion," recognizing that the busy state is the normal state.
• Finds support in being around like-minded people who care about AI safety.
• Values Anthropic's egoless culture and talent density.

17. Anthropic's Evolution and Ben's Roles:

• Ben has held about 15 different roles at Anthropic, from head of security to product lead.
• His favorite role has been leading the labs team (now Frontiers), which focuses on transferring research to end-user products.
• The labs team aims to differentiate Anthropic by being on the cutting edge and doing things that no other company can safely do.
• MCP (Model Context Protocol) and Claude Code came out of the labs team.

18. The Frontiers Team and "AGI-Pilled" Thinking:

• The Frontiers team works with the latest technologies and explores what is possible.
• The team is inspired by Google's Area 120 and Bell Labs.
• The team focuses on "skating to where the puck is going" by understanding the exponential and building for the future.
• The team is "AGI-pilled," meaning they are focused on building for a future where AGI is a reality.

Conclusion:

Ben Mann's insights reveal a deep commitment to AI safety and alignment, driven by a long-term vision of the potential benefits and risks of superintelligence. Anthropic's mission-driven culture, focus on constitutional AI, and emphasis on empirical research position it as a leader in responsible AI development. While the timeline for superintelligence remains uncertain, the need for proactive safety measures and a society-wide conversation about AI values is clear. The key takeaways are the importance of AI safety research, the need for better benchmarks, and the potential for AI to transform the economy and society.