Amazon’s AI Roadmap With AWS CEO Garman

Key Concepts

AI, Generative AI, AWS, Amazon Bedrock, Amazon Queue, Inference, Training, Tokenization, Project Right, Anthropic, Claude, Trainium, NVIDIA, GPU, P6 Instances, European Sovereign Cloud, Data Sovereignty.

Main Topics and Key Points

AWS's AI Business and Growth

• Multi-billion Dollar Run Rate: AWS's AI business has reached a multi-billion dollar run rate, encompassing customer-built models, Amazon Bedrock (hosted models like Amazon Nova and third-party models), and AI applications like Amazon Queue.
• AI Transformation: Every business, industry, and job will be fundamentally transformed by AI. The current multi-billion dollar business is just the beginning.
• Generative AI Revenue: The multi-billion dollar revenue is specifically for customers using AWS. Amazon also uses generative AI internally to optimize fulfillment centers, summarize reviews on the retail site, and enhance Alexa.
• Customer Applications: Customers are using AWS AI to transform contact centers (Amazon Connect) and build models using custom chips or NVIDIA processors.

Training vs. Inference Workloads

• Shifting Balance: Initially, AI usage was dominated by training, but now inference is becoming more prevalent.
• Future Prediction: Over time, 80-90% of AI workloads will be inference-based.
• Inference as a Building Block: Inference is becoming a core building block in every application, similar to compute, storage, and databases.
• Embedded AI: AI is increasingly embedded in applications, making it difficult to isolate AI-driven revenue.
• Current State: Inference usage is currently more than training.

Tokenization as a Metric

• Limited Measure: Token growth is a metric to consider, especially for text generation, but it's not the only one.
• Reasoning Models: For reasoning models, input and output tokens don't fully represent the work done. Models can "think" for hours before outputting tokens.
• Coding Example: In coding (e.g., Q Developer), the model iterates and improves itself, making the final output token an inadequate measure of the work done.
• Images and Videos: Token count doesn't capture the content creation and thought involved in images and videos.

Project Right and Anthropic Collaboration

• Largest Compute Cluster: Project Right is a collaboration with Anthropic to build the largest compute cluster for training the next generation of Claude models.
• Claude Adoption: Claude models, including the recently launched Claude 4, are experiencing significant adoption.
• Trainium2: Anthropic will train its next model on Trainium2, Amazon's custom-built accelerator processors for AI workloads.
• Operational Status: Training is landing on Trainium2 servers, and Anthropic is already using parts of the cluster.
• Performance: Trainium2 delivers impressive performance, pushing the boundaries of what's possible in terms of absolute performance, cost performance, and scale.

Cost of AI and Innovation

• Cost Concerns: AI is still considered too expensive.
• Cost Reduction Strategies: Innovation at the silicon level (e.g., Trainium) and on the software/algorithmic side is needed to reduce costs.
• Compute Efficiency: Reducing the compute required per unit of inference or training is crucial.

NVIDIA and Trainium

• Complementary Platforms: NVIDIA and Trainium are not seen as competitors but as complementary platforms. There is room for both.
• NVIDIA's Strengths: NVIDIA has a strong, leading platform for many applications.
• Design Partnership: AWS is a design partner with NVIDIA, offering the latest NVIDIA technology.
• Customer Choice: Customers want choice and shouldn't be forced into using one platform.
• AI Lab Interest: Leading AI labs are interested in using Trainium2.

NVIDIA GB200 and P6 Instances

• P6 Instance Availability: P6 instances, backed by NVIDIA GB200, are available on AWS.
• Capacity Ramping: AWS is aggressively ramping up capacity for P6 instances.
• Strong Demand: Demand for P6 instances is strong.

Anthropic's Model Availability on Other Platforms

• Acceptance of Multi-Cloud: AWS accepts that Anthropic's models are available on other platforms like Azure Foundry.
• AWS's Focus: AWS aims to be the best place to run every type of workload, including Anthropic's models.
• Customer Migration: Customers like Mondelez are migrating to AWS for cost optimization, availability, and security.
• Legacy Transformation: Mondelez is transforming legacy Windows platforms into Linux applications to save on licensing costs.
• Technical Capability: AWS strives to be the most technically capable platform with the widest set of services.

Data Center Expansion

• Latin America: Expanding capacity aggressively, with a new region in Mexico and announced region in Chile, plus an existing region in Brazil.
• Europe: Expanding in Europe, with the upcoming launch of the European Sovereign Cloud.
• European Sovereign Cloud: A unique capability designed for critical EU-focused sovereign workloads, addressing data sovereignty concerns for government and regulated workloads.

Important Examples, Case Studies, or Real-World Applications Discussed

• Amazon's Internal AI Usage: Optimizing fulfillment centers, summarizing reviews on the retail site, and enhancing Alexa.
• Amazon Connect: Transforming contact centers with AI capabilities.
• Q Developer: AI-powered coding assistant that iterates and improves code.
• Mondelez: Migrating to AWS and transforming legacy Windows platforms to Linux for cost savings.

Step-by-Step Processes, Methodologies, or Frameworks Explained

• AI Model Development and Deployment: The discussion touches on the entire lifecycle, from training large models to deploying them for inference in various applications.
• Cost Reduction in AI: The need for innovation at the silicon level and algorithmic level to reduce the cost of AI training and inference.

Key Arguments or Perspectives Presented, with Their Supporting Evidence

• AI's Transformative Potential: AI will fundamentally transform every business, industry, and job. Evidence: AWS's multi-billion dollar AI business, customer adoption of AI technologies.
• Inference Will Dominate AI Workloads: Over time, inference will account for the vast majority of AI workloads. Evidence: The increasing embedding of AI in applications.
• Tokenization is an Incomplete Metric: Token count is not a comprehensive measure of AI workload. Evidence: Reasoning models and image/video generation involve significant processing beyond token output.
• NVIDIA and Trainium are Complementary: There is room for both NVIDIA and Trainium in the AI market. Evidence: NVIDIA's established platform and AWS's focus on customer choice.

Notable Quotes or Significant Statements with Proper Attribution

• "Every single business, every single industry and really every single job is going to be fundamentally transformed by AI." - Matt Garman
• "Inference is a core building block. It's just like compute, it's just like storage. It's just like a database." - Matt Garman
• "Customers want choice. At the end of the day, customers don't want to be forced into using one platform or the other." - Matt Garman

Technical Terms, Concepts, or Specialized Vocabulary with Brief Explanations

• AI (Artificial Intelligence): The simulation of human intelligence processes by computer systems.
• Generative AI: A type of AI that can generate new content, such as text, images, or code.
• AWS (Amazon Web Services): Amazon's cloud computing platform.
• Amazon Bedrock: AWS's service for accessing foundation models from various providers.
• Amazon Queue: An AWS service for building AI-powered applications.
• Inference: The process of using a trained AI model to make predictions or decisions on new data.
• Training: The process of teaching an AI model to learn from data.
• Tokenization: The process of breaking down text into smaller units (tokens) for processing by AI models.
• Project Right: A collaboration between AWS and Anthropic to build a large compute cluster for AI training.
• Anthropic: An AI safety and research company.
• Claude: Anthropic's AI assistant.
• Trainium: Amazon's custom-built AI accelerator chip.
• NVIDIA: A leading manufacturer of GPUs (Graphics Processing Units) used for AI and other compute-intensive tasks.
• GPU (Graphics Processing Unit): A specialized processor designed for parallel processing, commonly used for AI training and inference.
• P6 Instances: AWS instances powered by NVIDIA GPUs.
• European Sovereign Cloud: An AWS cloud region designed to meet the data sovereignty requirements of European customers.
• Data Sovereignty: The concept that data is subject to the laws and regulations of the country in which it is located.

Logical Connections Between Different Sections and Ideas

The discussion flows logically from AWS's overall AI strategy and revenue to specific details about training vs. inference workloads, the importance of cost reduction, and the collaboration with Anthropic on Project Right. It then addresses the relationship with NVIDIA and the availability of their GPUs on AWS, before concluding with data center expansion plans and the European Sovereign Cloud initiative.

Data, Research Findings, or Statistics Mentioned

• AWS's AI business has a multi-billion dollar run rate.
• Anthropic's Claude 4 model has seen incredible adoption.
• Project Right's compute cluster is five times larger than the previous one used to train Claude.

Brief Synthesis/Conclusion of the Main Takeaways

AWS is heavily invested in AI and sees it as a transformative force across all industries. The company is focused on providing a comprehensive AI platform, offering both training and inference capabilities, and working with partners like Anthropic and NVIDIA to drive innovation and reduce costs. AWS is also committed to meeting the data sovereignty needs of its customers through initiatives like the European Sovereign Cloud. The key takeaway is that AWS aims to be the leading platform for AI, offering customers choice, performance, and security.