Who Governs AI? Inside the Rise of a New Empire

Key Concepts

• AGI (Artificial General Intelligence): Hypothetical AI with human-level cognitive abilities.
• Empire of AI: Karen How's concept describing companies like OpenAI as modern empires due to resource exploitation, labor practices, knowledge monopolization, and a "civilizing mission" narrative.
• Boomers vs. Doomers: Two factions within the AI world. Boomers believe AGI will bring utopia, while Doomers believe it will destroy humanity.
• Participatory AI Development: An approach advocating for collective ownership and public discussion throughout the AI development supply chain.
• Techno-Authoritarianism: The idea that Silicon Valley companies are developing technologies without democratic participation or consideration of societal impact.
• Scale at All Costs: Silicon Valley's approach to AI development, prioritizing rapid scaling and expansion of computing infrastructure, often with negative consequences.

Origin and Evolution of OpenAI

• Initial Nonprofit Conception: OpenAI was co-founded by Sam Altman and Elon Musk as a nonprofit to counter Google's AI dominance and develop AI for the public interest. Musk was concerned about AI being developed in a for-profit environment.
• Shift to For-Profit: In 2017, OpenAI realized that scaling AI development required vast capital (tens of billions). They nested a for-profit within the nonprofit to attract VC investment, initially with a "capped profit" model (100x return, later reduced).
• Musk's Departure: Altman and Musk disagreed on who should be CEO of the for-profit entity. The other co-founders (Greg Brockman and Ilya Sutskever) initially chose Musk but were later convinced by Altman to support him. Musk left, objecting to the for-profit shift and lack of control.
• Legal Challenge: The restructuring of OpenAI has faced legal challenges, highlighting the complexities of its governance.

The Rift Between Musk and Altman

• CEO Ambitions: Both Altman and Musk wanted to be CEO of the for-profit entity, leading to a power struggle.
• Brockman and Sutskever's Role: Brockman and Sutskever initially favored Musk but were swayed by Altman's appeal and concerns about Musk's trustworthiness.
• Musk's Continued Influence: Even after leaving, Musk maintained influence through Siobhan Zillis, who served on OpenAI's board.

Empire of AI: Key Characteristics

• Resource Exploitation: Claiming resources (data scraped from the internet) as their own, often without informed consent.
• Labor Exploitation: Contracting workers in the global south for data annotation and content moderation at low wages ($2/hour) and in psychologically toxic conditions. Also, aiming to create labor-automating machines.
• Monopolization of Knowledge Production: Dominating AI research, shifting AI researchers from academia to corporate environments, filtering research through corporate interests.
• "Good Empire" Narrative: Justifying their actions with a "civilizing mission" to bring progress and modernity to humanity.

OpenAI's Mission and Fiduciary Duty

• "Benefit All of Humanity": OpenAI's mission lacks clear definition, leading to subjective interpretations and serving as a narrative empire-building tool. There is no agreement within OpenAI on what "to ensure" means, what AGI is, or what "benefit to all of humanity" entails.
• Fiduciary Duty to Humanity: The concept of a fiduciary duty to humanity is legally meaningless but used rhetorically.

The Ouster and Reinstatement of Sam Altman

• Board's Concerns: The board, leaning towards the "Doomer" perspective, lost confidence in Altman due to his loose relationship with the truth and concerns about safety checks being skipped.
• Executive Concerns: Ilya Sutskever and Mira Murati independently approached the board with concerns about Altman's behavior and corporate governance.
• Governance vs. Economics: The ouster highlighted the tension between governance and economics, as employees initially sided with Altman due to a pending tender offer.
• Microsoft's Role: Microsoft attempted to acquire the company, but Altman was eventually reinstated.
• Investor Pressure: Investors have since pushed for restructuring to establish clear fiduciary responsibility to shareholders.

Departures of Key Figures

• Ego, Ideology, and Personality Clashes: Departures of co-founders and executives (e.g., Dario Amodei, Ilya Sutskever) stemmed from disagreements on how to shape AI and a desire to create AI in their own image.
• Public Benefit Corporations: Many of these new companies are structured as public benefit corporations, espousing a fiduciary duty to humanity.

AI Development in China

• Regulation: China is heavily regulating AI, second only to the EU, including data privacy and cybersecurity laws.
• "What About China" Card: US companies often use the "what about China" argument to ward off regulation, but China is already implementing AI regulations.
• Techno-Authoritarianism: The speaker argues that the "what about China" card has led to more authoritarian technologies in the US.

The Problem of "Scale at All Costs"

• Energy Demand: Rapid scaling of AI requires massive energy consumption, potentially leading to increased reliance on fossil fuels.
• Environmental Impact: Data center expansion is leading to the extension of coal plants and the use of unlicensed methane gas power plants, causing air pollution.
• Not Leading to a Good Place: The current trajectory of scaling at all costs is not sustainable or beneficial.

Ethically Developing AI: A Vision

• Participatory AI Development: Involving the public in discussions about data usage, data center placement, and AI deployment.
• Collective Ownership: Recognizing that the resources and spaces used for AI development are collectively owned.
• Chilean Water Activists Example: A community in Chile successfully blocked a Google data center to protect their freshwater supply, demonstrating the power of collective action.

Alignment Faking and the Nature of AI

• Language and Deceit: The speaker references Joseph Weizenbaum's work, highlighting the human tendency to ascribe intent to technologies that are ultimately just spewing language.
• Humans Acting Like Machines: The pursuit of replicating human intelligence may lead to humans increasingly acting like machines.

OpenAI's Progress and Marketing

• Asymptotic Research Progress: OpenAI's research capabilities are plateauing as they approach the limits of their scaling paradigm.
• Focus on User Interface: OpenAI is now prioritizing user interface improvements to maintain market dominance rather than competing based on research.
• Marketing vs. Reality: OpenAI continues to market itself as making rapid research progress, despite the reality of its asymptotic progress.

Open Source vs. Closed Source

• Monopolization of Knowledge Production: Closed-source models contribute to the monopolization of knowledge production.
• Need for Openness: Open-source models allow independent researchers to scrutinize the models and data sets, verifying their capabilities and limitations.
• Data Transparency: The speaker emphasizes the importance of data transparency to accurately evaluate AI models.

Public Policy and Countering AI Empires

• Funding Independent Research: Government should fund more research outside of corporate empires, including academia, nonprofits, and smaller companies.
• Resourcing Human Rights and Labor Rights Organizations: Supporting these organizations to participate in AI development.
• Avoiding New Empires: The goal should not be to create another "good empire" but to fund diverse organizations with independent expertise.

Leading the Discussion on AI

• Everyone's Responsibility: Everyone should be involved in discussions about AI and contribute to the conversation.
• Student Activism: Student activism and coalition building can play a role in shaping AI development.
• Stanford's Influence: Stanford is uniquely positioned to influence the culture that creates these technologies.

Recommendations for AI Startups

• Avoid Competing on Large-Scale Models: AI startups should not try to compete with large companies on large-scale AI models.
• Create New Imaginations of AI: Focus on developing narrowly scoped AI systems for specific applications, such as material science discovery.

Hopes and Fears for the Future of AI

• Hope: Increased public discourse and governance of AI development, leading to more broadly beneficial AI.
• Fear: Erosion of democracy due to a loss of agency and self-determination, leading to people giving up on their ability to shape the future.

Conclusion

The discussion highlights the complex issues surrounding AI development, particularly the rise of powerful AI companies and their impact on society. Karen How's "Empire of AI" framework provides a critical lens for understanding these companies' practices and motivations. The conversation emphasizes the need for participatory AI development, transparency, and a shift away from the "scale at all costs" paradigm to ensure that AI benefits all of humanity and does not erode democratic values.