The path to AI inferencing on GKE Part 1: Guided model research

Key Concepts

• AI Serving Platform: A system designed to deploy and manage artificial intelligence models for real-time use.
• Inference Quickstart Service (Google Cloud): A service that simplifies the deployment of AI models by providing a robust API and pre-configured tools.
• Chatbot Arena: A benchmark platform for Large Language Models (LLMs) that uses anonymous, randomized battles to rate model performance.
• ELO Rating: A rating system, commonly used in chess, adapted here to rank LLMs based on their performance in head-to-head comparisons.
• Committed Use Discounts (CUDs): Pricing options on Google Cloud that offer lower costs for committing to a certain level of usage over one or three years.
• On-Demand Pricing: Standard pricing for cloud resources, offering flexibility without long-term commitment.
• Spot Pricing: Significantly discounted pricing for unused cloud capacity, suitable for fault-tolerant workloads.
• Price per Million Output Tokens: A cost metric used to evaluate the expense of generating a million output tokens from a model.
• VLLM: An open-source model server platform optimized for high-throughput and low-latency LLM inference.
• Max Normalized Time Per Output Token (NTPOT): A performance metric representing the normalized time taken to generate each output token.
• Max Time to First Token (TTFT): A performance metric measuring the time it takes for a model to generate the very first token of its response.
• Cost per Million Input and Output Tokens: A comprehensive cost metric that considers both the tokens used for prompts and the tokens generated as output.
• Throughput vs. Latency: Key performance indicators where throughput measures the number of requests processed per unit of time, and latency measures the time taken for a single request.
• NVIDIA GPU (A3 high GPU 1G instance): A specific type of high-performance graphics processing unit available on Google Cloud, used for accelerating AI workloads.
• GKE MCP Server (Gemini CLI): A component that allows interaction with Google Kubernetes Engine (GKE) through natural language commands via the Gemini CLI.
• Gemini CLI: A command-line interface powered by Gemini models, enabling natural language interaction with Google Cloud services.
• Kubernetes Manifest: A configuration file (typically YAML) that describes the desired state of Kubernetes resources, such as deployments and pods.
• Pod Monitoring Specification: Configuration within a Kubernetes manifest to collect metrics from pods, often for services like Google Managed Prometheus.
• Horizontal Pod Autoscaler (HPA): A Kubernetes feature that automatically scales the number of pods in a deployment based on observed metrics.
• GPU Cache Usage Percentage: A metric used for autoscaling, indicating how much of the GPU's cache memory is being utilized.
• Time to Market: The duration from product conception to its availability to customers.

Model Selection and Benchmarking

The process of building a robust AI serving platform begins with selecting the appropriate model for a specific use case. AI researchers can leverage Google Cloud documentation for best practices. The recent launch of Google Cloud's Inference Quickstart Service, with its robust API and accompanying Google Colab notebook, significantly aids this process.

After setting up and authenticating to a Google Cloud project, the researcher can utilize a Colab notebook to create a comparative table. This table juxtaposes Google-benchmarked models with their ELO ratings from Chatbot Arena. Chatbot Arena is a benchmark platform for LLMs that employs anonymous, randomized battles to assess model performance. The notebook also integrates with the billing API, allowing the researcher to select optimal pricing models, including one- and three-year committed use discounts, on-demand, and spot pricing.

A key visualization presented is a chart comparing various open-source models. This chart illustrates the relative intelligence of each model against a common metric in AI serving: the cost per million output tokens. This data-driven approach helps narrow down model choices, identifying those that can serve the most use cases cost-effectively. For instance, while GPOSS20B offers low cost and a good ELO rating, the researcher might opt for a smaller model like Metal Lama 38B based on specific project needs.

Technical Model Serving Configuration

Once a model is tentatively selected, technical specifics regarding its serving are determined. This includes choosing a model server platform, such as VLLM, and verifying the specific version that was benchmarked. This verified version is then incorporated as a variable within the notebook.

The notebook provides functionality to simulate different model and performance goals, enabling visualization of Google's benchmarked data. Key performance metrics like Max Normalized Time Per Output Token (NTPOT) and Max Time to First Token (TTFT), along with cost metrics such as cost per million input and output tokens, are crucial for narrowing down selections of models, accelerators, and machine types.

Performance and Cost Analysis

The notebook can generate various charts to examine different dimensions of the benchmarked data. An example provided shows throughput versus latency for a selected model across different benchmarked machine types, where each machine type on Google Cloud corresponds to a specific NVIDIA GPU. This analysis quickly reveals that the A3 high GPU 1G instance offers the optimal performance profile.

This capability allows researchers to identify the necessary components to meet performance and/or cost goals. Since the notebook is Python-based, researchers can integrate their own datasets to further refine their decisions. This empowers them to provide data-driven recommendations to their platform team for infrastructure provisioning.

Infrastructure Provisioning with Gemini CLI

Platform engineers can then utilize the researcher's findings. By employing the GKE MCP server within the Gemini CLI, they can verify and interact with other platform and development tools through a natural language interface to build the required infrastructure.

The engineer can verify the researcher's findings directly against the Inference Quickstart API using the CLI. This data serves as context for the engineer's subsequent prompt to generate a Kubernetes manifest for the model, model server, server version, and chosen accelerator. The Gemini CLI, powered by Gemini models, intelligently uses the GKE MCP server to call the Inference Quickstart API and retrieve the manifest prescribed by the service. The GKE MCP server then determines the optimal Kubernetes manifest for a deployment based on the benchmarked profile.

Through the Gemini CLI's interactive interface, the engineer can instruct Gemini to save the manifest file to local storage and perform other operations before committing it to their production CI/CD or IA platform.

Kubernetes Manifest Details

A closer examination of the provided manifest reveals the specifics of what the Inference Quickstart API delivers. The manifest includes a pod monitoring specification for VLLM to output metrics to Google Managed Prometheus. A Horizontal Pod Autoscaler (HPA) is also created, targeting the VLLM metric "GPU cache usage percentage." The deployment itself incorporates key VLLM server arguments used during the benchmarking process, ensuring the model server is tuned correctly.

This pre-configuration significantly reduces post-deployment tuning efforts for AI engineers and platform teams, leading to a faster time to market for critical organizational projects.

Conclusion

The GKE Inference Quickstart is presented as the initial step in a streamlined path to production AI serving on Google Kubernetes Engine and Google Cloud. The video concludes by inviting viewers to explore further resources.