Connecting Your AI Agent to a Cloud-Hosted LLM

Key Concepts

• GPU-accelerated AI brain: A powerful artificial intelligence model running on Graphics Processing Units (GPUs) for enhanced performance.
• Cloud Run: A managed compute platform that allows you to run stateless containers.
• Gemma: A family of lightweight, state-of-the-art open models developed by Google.
• Agent Development Kit (ADK): A framework or library used to build AI agents, specifically for conversational logic.
• LiteLLM: A library that provides a unified interface to connect to hundreds of different Large Language Model (LLM) APIs.
• Ollama: An open-source tool that enables running large language models locally.
• Ollama-compatible API: An API that adheres to the specifications of Ollama, allowing for interoperability.
• Decoupling: Designing systems so that different components can operate and scale independently.
• Session state: Information maintained for a specific user interaction or session.
• Environment variables: Variables that can be set in the operating system or container environment to configure applications.

Building a Conversational AI Agent with ADK and Gemma

This video details the process of building and deploying a conversational AI agent that interacts with a pre-deployed GPU-accelerated LLM (Gemma). The core objective is to create a separate agent service that can communicate with the LLM backend, enabling independent scaling and development.

1. Agent Architecture and Core Components

• Decoupled Services: The system is designed with two distinct services:
  • GPU Service: Hosts the powerful, GPU-accelerated LLM (Gemma in this case). This service is resource-intensive, requiring GPUs.
  • Agent Service: Manages conversational logic, user interaction, and session state. This service is lightweight and does not require GPUs.
• Agent Development Kit (ADK): This kit is used to build the agent's conversational logic. The primary file is agent.py.
• LiteLLM Integration: The ADK leverages LiteLLM to connect to various LLM APIs. This allows for easy switching of models by simply changing a configuration string.

2. Configuring the ADK Agent

• agent.py File: This file contains the core logic for the agent.
• Model Parameter Configuration: The model parameter within the ADK is crucial. It uses a string to define the LLM to be used:
  • "ollama/gemma:270m": This specific string tells ADK to connect to an Ollama-compatible API, use a chat interface, and specifically request the "Gemma 3 270M" model.
• Prompt Engineering: A standard prompt is used to define the agent's persona and role. In this example, the agent is configured as "Gem," a friendly zoo tour guide.

3. Deployment of the Agent Service

• Separate Dockerfile: The agent service has its own Dockerfile for dependency management.
• Resource Allocation: The deployment command for the agent service is configured for significantly less resources compared to the LLM service:
  • Low Memory and CPU: The agent service is lightweight, focusing on request handling and routing.
  • No GPU Required: As it doesn't perform LLM inference, it does not need GPU acceleration.
• Environment Variables:
  • OLLAMA_API_BASE: This is a critical environment variable. It is set to the URL of the deployed Gemma LLM service. This allows the agent service to forward user requests to the correct backend LLM.

4. Communication Flow and Real-World Application

• User Interaction: A user sends a message through the agent's web UI.
• Agent Forwarding: The agent service receives the message.
• LLM Inference: The agent service, using LiteLLM and the configured OLLAMA_API_BASE, forwards the request to the Gemma GPU service.
• Response Generation: The Gemma LLM processes the request and generates a response.
• Response Delivery: The response is sent back through the agent service to the user.

5. Example Use Case: Zoo Tour Guide

• Test Case 1: "What do red pandas typically eat in the wild?"
  • The agent successfully queries the Gemma model, which provides an accurate answer.
• Test Case 2: "Why are poison dart frogs so brightly colored?"
  • Another successful interaction, demonstrating the agent's ability to handle diverse queries.

6. Production Readiness and Future Steps

• Scalability Challenge: The current setup is functional for single users but needs to be tested under heavy load.
• Next Steps: The video announces that the next installment will focus on simulating a massive traffic spike to observe the system's automatic scaling capabilities.

Conclusion

This video successfully demonstrates the creation of a production-style AI agent by decoupling the conversational logic from the LLM inference engine. By utilizing the ADK, LiteLLM, and Cloud Run, a flexible and scalable architecture is established. The agent service acts as an intelligent intermediary, routing requests to a powerful, GPU-accelerated Gemma model, paving the way for robust AI applications. The key takeaway is the ability to build and deploy independent services that communicate effectively, allowing for optimized resource utilization and independent scaling.