Optimize model serving with GKE Inference Gateway

GKE Inference Gateway: Optimizing Gen AI Workloads on Kubernetes

Key Concepts:

• GKE Inference Gateway: An enhancement to the existing GKE Gateway designed to optimize AI inference workloads, particularly LLMs, on Kubernetes.
• LLM Workload Characteristics: High variability in processing time, computationally intensive requests, and unique performance requirements (latency vs. throughput).
• KV Cache Utilization: A metric used to determine the least utilized GPU for routing requests, improving performance and reducing latency.
• Multi-Region GPU/TPU Pooling: Ability to pull GPU/TPU capacity from multiple Google Cloud regions to address capacity constraints.
• AI Safety and Security Guardrails: Embedded security measures within the gateway to protect against attacks like prompt injections and data leaks.
• Model Armor: Google Cloud's AI safety and security service integrated with GKE Inference Gateway.
• Kubernetes Gateway API: The foundation upon which GKE Inference Gateway is built, leveraging extensions for AI inference-specific functionalities.

1. Challenges of Running LLMs on Kubernetes:

• GPU/TPU Capacity Constraints: Limited availability of GPU/TPU infrastructure across regions and providers.
• Unique Workload Profiles: LLM workloads differ significantly from traditional web workloads, requiring specialized traffic distribution and resource allocation strategies.
• Compute Allocation: Determining the optimal amount of compute to allocate for each model is crucial due to the high cost of GPUs/TPUs.

2. GKE Inference Gateway: A Solution:

• Enhancement, Not a New Product: GKE Inference Gateway is an improvement to the existing GKE Gateway, available in public preview.
• Key Features:
  • Enhanced load balancing algorithm tailored for LLM workloads.
  • Autoscaling based on metrics from model servers.
  • Multi-LoRa model packing on common GPU/TPU pools.
  • Multi-region GPU/TPU capacity pooling.
  • Embedded AI safety and security guardrails.
  • Enhanced cloud monitoring dashboards.

3. Enhanced Load Balancing for LLMs:

• Problem with Round Robin Routing: High variability in processing time can lead to oversubscription of GPUs, resulting in high wait times and poor user experience.
• KV Cache Utilization as a Metric: KV cache utilization and pending requests provide the best signal for identifying the least utilized GPU.
• Benefits: Improved performance and lower latency by routing traffic to the most available GPU.

4. Real-World Example: Llama 2 Model on H100s:

• Setup: Llama 2 model served using vLLM on H100 GPUs, with six model replicas and an increasing QPS load (100-200 QPS).
• Traditional Load Balancing Issues: Variability in KV cache utilization across replicas, with some replicas becoming saturated and requests queuing up.
• GKE Inference Gateway Performance: Consistent KV cache utilization across all replicas, with no queuing observed.
• Performance Improvements: Lower and more predictable latency, higher throughput, and ability to handle heavier QPS load without performance degradation.

5. Multi-Region GPU/TPU Pooling:

• Problem: Regional capacity issues can disrupt AI services.
• Solution: GKE Inference Gateway automatically routes requests to alternative regions with available capacity.
• Benefits: High-scale AI services that are resilient to demand surges and regional capacity issues.

6. Serving Diverse Model Workloads:

• Different Performance Requirements: Chatbots are latency-sensitive, while reasoning models prioritize throughput.
• GKE Inference Gateway Prioritization: Allows prioritizing traffic based on model type (e.g., chatbots over reasoning models).
• Benefits: Ability to serve multiple families of models with the appropriate user experience.

7. Open and Extensible Architecture:

• Foundation: Kubernetes Gateway API and Google Cloud's Load Balancing.
• Enhancements:
  • Routing based on the model name in the request body (OpenAI API spec).
  • Integration with AI safety and security services (e.g., Google Cloud Model Armor).
  • Polling metrics from model servers (KV cache utilization, queue length) to determine optimal routing.

8. AI Safety and Security:

• LLMs as Attack Surfaces: Susceptible to jailbreaking, prompt injections, data leaks, and harmful content generation.
• Embedded Guardrails: GKE Inference Gateway embeds AI security and safety measures at the gateway level.
• Integrations: Google Cloud Model Armor, Palo Alto Networks AI Runtime Security Service, and NEMO guardrails.
• Benefits: Consistent protection and centralized management of AI safety and security across all models.

9. Conclusion:

GKE Inference Gateway provides significant enhancements for running Gen AI workloads on Kubernetes. By optimizing load balancing, enabling multi-region GPU/TPU pooling, and embedding AI safety and security guardrails, it helps organizations achieve better price-performance, improved user experience, and greater control over their AI deployments. The key takeaway is that GKE Inference Gateway addresses the unique challenges of LLM workloads, allowing for more efficient and secure utilization of expensive GPU/TPU resources.