Google Cloud Therapist on Bringing AI to Cloud Native Infrastructure

Key Concepts

Cloud Therapist, GKE (Google Kubernetes Engine), AI (Artificial Intelligence), Vert.ai, Hardware Accelerators (GPUs, TPUs), Model as a Service, GKE Inference Gateway, Inference Quick Start, Resource Fungeibility, Dynamic Workload Scheduler (DWS), Custom Compute Class (CCC), LLM (Large Language Model), Agents, Model Routing, Load Balancing, Kubernetes, Cloud Native, Training, Inference, Model Evaluation, Heterogeneous Environments.

Cloud Therapist Philosophy

Bobby Allen introduces the concept of "Cloud Therapist" as a role that prioritizes listening to customer problems over pushing specific Google products. He emphasizes that technology is the easy part of tech, while people, behavior, and humility are more challenging aspects. He believes in respecting the history and existing infrastructure of customers before introducing new technologies.

Quote: "Technology is the easy part of tech, people are the best part, behavior is the hard part, humility is the worst part."

GKE and AI Integration

GKE is presented as a foundational platform for AI, both directly and indirectly.

• Indirectly: GKE powers services like Vertex AI, benefiting from the scale and robustness requirements of internal customers like DeepMind.
• Directly: Many customers run AI training and inference workloads on GKE, leveraging its orchestration capabilities.
• GKE allows for tuning and customization to meet specific use cases, acting as a primitive that underlies various AI applications.

Evolution of AI on GKE

The discussion highlights the shift from commoditized hardware back to specialized hardware (GPUs, TPUs) for AI workloads. GKE acts as an orchestrator to efficiently manage these specialized resources. AI is viewed as a modern, scalable, and elastic workload with unique characteristics, such as the need for hardware accelerators. Kubernetes provides the primitives needed to configure and control AI environments.

AI Adoption Stage

AI is described as being in its early stages of adoption, comparable to a toddler. This suggests significant opportunities for newcomers to contribute and shape the field. The importance of remixing new technologies with proven practices from the past is emphasized.

Quote: "Everything new isn't good, and everything old isn't bad."

GKE Modes and AI Workloads

GKE offers different modes to cater to varying customer needs:

• GKE Standard: Provides full control and customization options.
• GKE Autopilot: Offers sensible defaults and automatic configuration.

Customers can run AI workloads on either mode, depending on their desired level of control and management.

AI Value Proposition

The discussion categorizes AI into three broad areas:

• What you build: Training, inference, and fine-tuning of AI models.
• How you build: Integrating AI into the software development lifecycle.
• How you operate: Using AI to optimize platform operations (e.g., cost optimization, security gap identification, troubleshooting).

The "how you operate" category is identified as a low-hanging fruit opportunity for many organizations.

GKE's Role in the AI Platform Landscape

GKE is positioned as offering the most control with the least technical debt, providing scalability and future-proofing. While Vertex AI offers a batteries-included approach, GKE allows for greater customization and flexibility. Customers may choose multiple platforms (GKE, Vertex AI, Cloud Run) based on their specific needs and roles within the organization.

AI as a Driver for GKE Development

AI is a significant driver for GKE's product roadmap. However, it's acknowledged that many customers are still in the early stages of AI adoption. The focus is on enabling customers to experiment with models, evaluate their suitability, and expose them to their teams.

Model as a Service and GKE

The concept of "Model as a Service" is introduced, where GKE supports routing to models based on their purpose rather than their brand or version. This allows developers to access AI capabilities without needing to know the underlying infrastructure.

GKE Inference Capabilities

Two key features are highlighted:

• GKE Inference Quick Start: Provides prescriptive guidance and benchmarks for deploying models, including t-shirt sizes for resource allocation based on performance requirements (e.g., tokens per second).
• GKE Inference Gateway: Offers intelligent, LLM-aware routing to optimize model responsiveness. It prevents overloading resources with long-running requests and directs traffic to endpoints that can best serve the request.

Agents and GKE

GKE is one of several platforms for building agents, offering flexibility and control. Google Cloud aims to provide multiple on-ramps to cater to different customer preferences and use cases.

Customer Priorities for GKE

Customers are seeking:

• Hardware Accelerator Fungeibility: Easy switching between GPUs and TPUs.
• Obtainability: Ensuring resources are available when needed.
• Heterogeneous Environments: Running AI and non-AI workloads on the same platform.

Future Investments

Future investments will focus on:

• Best practices for orchestrating different types of workloads on GKE.
• Simplifying security, configuration, storage, and observability across workloads.
• Enabling customization and flexibility to accommodate diverse workload requirements.

Conclusion

GKE is a versatile and evolving platform that plays a crucial role in the AI landscape. By providing control, scalability, and flexibility, GKE empowers organizations to build, deploy, and manage AI workloads effectively. The focus on customer needs, hardware accelerator fungeibility, and intelligent routing mechanisms positions GKE as a key enabler of AI innovation.