Why CPUs still matter in the age of AI agents

Key Concepts

• Agentic AI: AI systems capable of autonomous task execution, tool usage, and code generation rather than just conversational responses.
• ARM Neoverse: A high-performance, energy-efficient CPU architecture designed for cloud workloads.
• Google Axion: Google Cloud’s custom ARM-based processors designed for high performance and cost-efficiency.
• GKE Agent Sandbox: A Google Kubernetes Engine (GKE) feature providing secure, isolated environments for running agentic workloads.
• gVisor: An open-source sandboxing technology that provides an isolation layer between applications and the host OS.
• Orchestration: The CPU’s role in managing agent workflows, API calls, and memory management.

---

1. The Evolving Role of CPUs in AI

While GPUs are typically associated with training large models, the speakers emphasize that CPUs are the "air traffic controllers" of the agentic AI era. As AI shifts from simple chatbots to autonomous agents that execute code, call tools, and process data, the demand for CPU-intensive orchestration increases.

• Key Functionality: CPUs handle concurrent, distributed workloads, memory management, and the execution of smaller, specialized models (e.g., 8 billion parameter models) that do not require the overhead of a GPU.
• Performance Metrics: ARM-based instances, specifically Google Axion, offer up to a 40% improvement in price-performance compared to traditional cloud alternatives.

2. Infrastructure and Frameworks

The discussion highlights a "portfolio approach" to infrastructure, utilizing specific processor types for different agentic needs:

• Axion C4A: Optimized for high single-threaded performance, ideal for complex orchestration and specialized model inference.
• Axion N4A: Optimized for cost and efficiency, well-suited for sandboxed environments.
• GKE Agent Sandbox: This framework allows developers to deploy agents at scale. It supports up to 300 sandboxes per second per cluster with a "time to first instruction" of less than one second.

3. Security and Isolation

Security is described as "non-negotiable" in agentic AI. Because agents often execute unverified or dynamic code, isolation is critical.

• gVisor Integration: By using gVisor, Google Cloud provides a secure isolation layer between the agent’s code and the host OS. This allows enterprises to run untrusted code safely within a managed environment.
• MicroVMs: Agents can programmatically spin up lightweight microVMs to perform tasks, ensuring that one agent’s activity does not compromise the host or other agents.

4. Cost Optimization Strategies

The speakers argue that efficiency in compute allows enterprises to reallocate budget toward "tokens" (model usage).

• Idle Time Management: Agentic workloads are often "spiky." Technologies like pod snapshots and warm pools are used to manage infrastructure costs during periods where agents are waiting for sub-agent responses or external API calls.
• Managed Services: Services like AlloyDB and Cloud SQL are natively supported on Axion, allowing for a seamless transition for existing cloud-native applications.

5. Implementation and Adoption

For developers looking to transition to ARM-based architectures, the speakers note that the ecosystem is now "thriving" and "mature."

• Porting Tools: AI-assisted tools, such as MCP (Model Context Protocol) servers, are available to guide developers in porting code from x86 to ARM architectures.
• One-Click Deployment: Through the Gemini enterprise agent platform, users can deploy models like Gemma directly onto Axion-powered infrastructure.
• Resources: Developers are encouraged to visit learn.arm.com for best practices and Axion-specific documentation.

Synthesis

The transition to agentic AI requires a shift in how infrastructure is viewed. While GPUs remain essential for model training, the operational backbone of agents—orchestration, security, and cost-effective execution—relies on high-performance CPUs. By leveraging ARM-based processors like Google Axion and cloud-native tools like GKE Agent Sandbox, enterprises can achieve the scale and security necessary to move agentic AI from pilot programs into production environments without compromising on performance or budget.