Scale Gov Summit 2024: Alexandr Wang Keynote

Key Concepts

AI, Algorithms, Compute, Data, Data Supremacy, AI Overmatch, AI-ready Data, Data Engine, Large Language Models (LLMs), Generative AI, Agent Workflows, Test and Evaluation Framework, National Security, US vs. China AI Competition, Power Consumption, STEM Talent, Donovan, Agent Donovan, RAG (Retrieval Augmented Generation), Frontier Data, AI Safety, AI Alignment.

US vs. China AI Competition

• Overall Assessment: The US is currently ahead in AI, but China is catching up rapidly and could surpass the US if current trends continue. Two years ago, China was significantly behind (more than two years), but now Chinese LLM and computer vision capabilities are nearly neck-and-neck with the US.
• LLMs: Chinese company Z1 produced Yarge, a model comparable to leading US models like GPT-4, Gemini, and Claude 3.5.
• Chips: Huawei's Ascend 910b chip, manufactured by SMIC, is about 80% as good as Nvidia's A100 but is two to three times more costly in terms of performance per dollar. While not immediately competitive with Nvidia's H100 or B200, they are approaching meaningful competitiveness.
• Power: The US is facing a critical vulnerability in power generation for AI data centers. The US electricity generation has grown by only 5% in the last decade, while AI's demand for electricity is outpacing production. China has added an entire US electrical grid's worth of power generation in the past decade and is projected to surpass the US in nuclear power capacity by 2030. China is likely 10-15 years ahead of the US on nuclear power.
• Talent: China produces more high-end STEM talent, but the US benefits from a "one-way road" where talent leaves China and stays in America.
• Data: The US and China have an uneven playing field regarding data usage. China utilizes data from the internet and overseas without the same legal constraints as the US. The CCP has built "data factories" where citizens produce data for AI models.

Three Pillars of AI

1. Algorithms: Advancements driven by AI research community, applied by top talent to develop models (GPT-4, Claude 3.5, Llama 3, etc.).
2. Compute: Advancements in chip manufacturing (Nvidia, Huawei).
3. Data: The speaker's company, Scale AI, focuses on providing the data that fuels AI development.

Achieving AI Overmatch: Three Steps

1. Achieve Data Supremacy:
   • AI success depends on building and maintaining data supremacy.
   • The internet is not enough; AI needs AI-ready data (expert data, labeled, tagged, and curated).
   • Even advanced commercial models are only as useful as the data they're trained on.
   • China benchmarks their data investments based on US commercial companies, which invest over $1 billion a year each.
   • The DOD plans to spend roughly $3 billion on AI overall, but only $100 million on AI data, while China spends at least $1 billion on AI-ready data.
   • The US government has vastly more data than was used to train GPT-4 (DOD generates roughly 360 petabytes of data per year, while GPT-4's training data was about 1 petabyte).
2. Invest to Beat China:
   • Prioritize investment in both AI systems and the underlying data.
3. Armor War Fighters with Best-in-Class Commercial AI Technology:
   • Leverage platforms like Donovan and data engines to give war fighters a competitive advantage.

Scale AI's Role and Products

• Mission: To help achieve US data supremacy.
• Data Engine: Powers major breakthroughs in AI, including fully autonomous driving and major AI programs within the US DOD.
• Donovan: An AI platform that allows government to use leading generative AI technologies securely with their own data. It was the first LLM deployed on DOD classified networks. Users have seen reporting overhead reduced from hours to minutes.
• Agent Donovan: The next evolution of the Donovan platform, featuring increasingly complex native agent workflows tailored for defense and national security. It brings new capabilities around reasoning and tool use, going beyond summarization and chat.
  • Example Workflow: Quickly locating reports of uranium munition strikes on Israel by connecting to data sources, identifying critical information, running code, correcting code execution errors, and validating sources.
• Data Foundry: Provides the most advanced and exquisite Frontier data.
• SEAL (Safety Evaluation and Alignment Lab): Develops authoritative, reliable evaluation sets for AI models.

Data Supremacy in Detail

• Data Engine Importance: Transforms raw data into AI-ready data.
• Data Volume: The Air Force's ISR PED activities generated roughly 22 terabytes of data daily in 2017. The DOD likely generates close to one petabyte a day now.
• Imagery Analysis: Without AI, the NGA would need over 8 million imagery analysts by 2027 to process all available imagery data.
• Data Investment Disparity: US commercial companies invest north of $1 billion a year each in data, while the DOD plans to spend no more than $100 million on AI data.

The Paradigm Shift in AI Capabilities

• From Simple Tasks to Complex Workflows: Moving beyond simple question-and-answer models to AI that can perform complex, multi-step tasks.
• Agent Workflows: Tailored for defense and national security, enabling AI to act as a partner and force multiplier.

Test and Evaluation Framework

• Importance: Understanding the strengths and limitations of AI systems.
• Testing Rigor: Should match the risk level of each use case.
• Scale AI's Role: Bringing technical expertise to the public sector to ensure models are deployed with the same level of rigor as commercial partners.
• Partnership with CDAO: Developing an LLM test and evaluation framework for the responsible use of LLMs within the DOD.

Notable Quotes

• "In AI we are in the mid game now...we're on the cusp of losing our momentum."
• "The history of war is the history of military technology and now everyone in this room understands that AI technology is the potential to be one of the greatest military assets that Mandy has ever seen."
• "AI always boils down to data."
• "If they continue to take data seriously and we don't, China will win."

Technical Terms and Concepts

• LLM (Large Language Model): A type of AI model trained on vast amounts of text data to generate human-like text.
• Generative AI: AI models that can generate new content, such as text, images, or code.
• AI-ready Data: Data that is properly labeled, tagged, and curated for use in training AI models.
• Data Engine: A system for transforming raw data into AI-ready data.
• RAG (Retrieval Augmented Generation): A technique for improving the accuracy and relevance of LLM-generated text by retrieving information from external sources.
• Frontier Data: Exquisite, high-end expert data sets that enable advanced AI capabilities.
• AI Safety: Ensuring that AI systems are safe, reliable, and aligned with human values.
• AI Alignment: Ensuring that AI systems' goals and behaviors are aligned with human intentions.

Synthesis/Conclusion

The US is in a critical race against China for AI supremacy. While currently ahead, the US risks falling behind due to China's rapid progress in algorithms, compute, power generation, and data utilization. To maintain its lead, the US must prioritize data supremacy by investing in AI-ready data, leveraging commercial AI technologies, and establishing robust test and evaluation frameworks. The development of advanced platforms like Donovan and Agent Donovan, powered by high-quality data and sophisticated AI models, is crucial for ensuring that the US military and national security agencies maintain a competitive edge. The speaker emphasizes that immediate action and collaborative efforts are essential to secure American AI leadership and ensure the Free World prevails.