Engineering the future of Kubernetes for AI at scale

Key Concepts

• Agentic Workloads: AI applications that use instructions, tools, and models to achieve goals.
• Dynamic Resource Allocation (DRA): A Kubernetes framework for granularly describing and managing specialized hardware (GPUs, TPUs, network interfaces).
• In-Place Pod Resize: A feature allowing resource modification (CPU/Memory) without restarting pods.
• Agent Sandbox: A tool for secure, isolated execution of untrusted code generated by AI agents.
• LMD (Large Model Distribution): A framework for stitching together models that exceed the capacity of a single node.
• Gateway API: The modern standard for managing ingress traffic in Kubernetes, replacing the legacy Ingress object.
• DRA-Net: An extension of DRA focused on managing physical networking resources (e.g., RDMA/TPU paths).

---

1. AI Applications and Agents

The speakers define an AI agent as a program that takes a dependency on a model, consisting of three core components:

• Instructions: Natural language prompts or markdown-based "skills" that guide behavior.
• Tools: Deterministic code (e.g., Python functions) used to perform specific tasks reliably, reducing hallucination.
• Models: The non-deterministic "math" that provides the AI's reasoning capabilities.

Key Features for Agents:

• Agent Sandbox: Provides a vendor-agnostic Custom Resource Definition (CRD) to isolate model-dependent workloads using runtimes like gVisor or Kata Containers. It includes "warm pools" to enable sub-second startup times for agent swarms.
• GKE Snapshots: Allows saving/restoring the state of a running pod. This is critical for rapidly spinning up model servers (by pre-loading weights) or pausing long-running batch jobs to free up GPUs.
• In-Place Resize: GA in Kubernetes 1.35, this allows non-disruptive scaling of resources. It enables "perfect bin packing," where pods can start with high resource requests for heavy imports and scale down to idle levels, saving costs.

---

2. Compute for Inference

As AI workloads shift from generic CPU/memory requirements to hardware-sensitive needs, Kubernetes is evolving to become "accelerator native."

• Dynamic Resource Allocation (DRA): Moves away from the "node is a node" abstraction. It allows workloads to express specific hardware requirements (e.g., specific GPU types), improving utilization of reserved capacity.
• LMD (Large Model Distribution): A framework for "disaggregated serving." It splits inference into prefill (processing the prompt) and decode (generating the response). It utilizes multi-node KV caching to optimize performance for models too large for a single node.
• Workload-Aware Scheduling: Incorporates concepts from projects like Kueue and Volcano into the core Kubernetes scheduler. Key capabilities include:
  • Gang Scheduling: Scheduling multiple pods simultaneously to prevent partial deployments that waste capacity.
  • Workload-Aware Preemption: Automatically evicting low-priority jobs to make room for high-priority AI tasks.
  • Topology-Aware Scheduling: Placing workloads based on physical proximity to high-speed networking and compute domains.

---

3. Networking for Inference

High-performance AI requires specialized networking to handle massive data synchronization.

• DRA-Net: Applies the DRA framework to networking. It allows for the assignment of dedicated high-speed RDMA or TPU network paths, which can improve bandwidth for distributed AI/ML workloads by up to 60%.
• GKE Inference Gateway: An implementation of the Gateway API. Unlike legacy Ingress, it moves routing logic to the load balancer. It integrates with LMD to perform "context-aware routing," using server metrics and prefix cache matches to route traffic to the node already holding the relevant KV cache.
  • Case Study: Vertex AI used this to double its prefix cache hit rate, reducing "time to first token" by 35% for StarCoder models and 52% for AppSec models.

---

4. Synthesis and Conclusion

The transition to AI at scale requires Kubernetes to evolve from a platform that abstracts hardware to one that manages it with extreme granularity. The industry is moving past the prototype phase (where only 7% of organizations deploy daily) into production-grade AI.

Main Takeaways:

1. Standardization: The community is pulling successful external projects (Kueue, Gateway API, etc.) into the Kubernetes core to ensure industry-wide consistency.
2. Efficiency: Features like In-Place Resize and GKE Snapshots are essential for managing the high costs of GPU/TPU resources.
3. Performance: Disaggregated serving (LMD) and specialized networking (DRA-Net) are the new requirements for competitive AI product delivery.

Quote: "You're no longer succeeding in spite of Kubernetes. You're succeeding because of Kubernetes." — Jay MacLeod