Hybrid LLMs: Utilizing Gemini and Gemma for edge AI applications

Key Concepts:

• Hybrid LLM architecture
• On-device LLMs (Gemma)
• Cloud-based LLMs (Gemini)
• Routing architecture
• Text classification (Mobile BERT)
• Local vs. Remote queries
• RAG (Retrieval-Augmented Generation)
• Evaluation metrics (quality, latency, labeling outcome)

1. Definition of Hybrid LLM Architecture

• A hybrid LLM architecture strategically distributes components like language models, data sources, and processing tools across different devices and servers.
• The goal is to optimize for performance, privacy, and model capabilities.

2. Model Choices: Gemini and Gemma

• The video focuses on using Gemini (Google's family of cloud-based multi-modal models) and Gemma (Google's family of open models capable of running on-device).
• Gemma models (specifically the 1B and 4B sizes) can run on mobile phones, tablets, and laptops with acceleration.
• Gemini 2.5 Pro is used as the cloud-based model for reasoning and real-time data access.

3. Reasons for Deploying a Hybrid LLM Architecture

• Leverage local compute resources for simple tasks.
• Utilize cloud compute for intensive tasks like reasoning or processing large data sources.
• Enable multimodal model usage offline.
• Operate on data that cannot leave the local device due to privacy or security constraints.

4. Architectural Options

• Sequential Architecture: On-device LLM reasons about a task and decides whether to send it to the cloud model.
• Parallel Architecture: Both models reason about the task, and their responses are merged.
• Routing Architecture: A text classifier decides whether to route a query to an on-device or cloud-based model. The video focuses on this architecture.

5. Routing Architecture in Detail

• Components:
  • On-device model: Gemma 3 1B (balances performance and capability on mobile devices).
  • Cloud-based model: Gemini 2.5 Pro (for reasoning and real-time data access).
  • Router: Mobile BERT (a text classifier optimized for mobile hardware).
• Process:
  1. Train the BERT-based router to label user queries as either "local" or "remote."
  2. Use a few thousand pre-labeled queries synthesized using the Gemma 3 27B model for training.
  3. Configure the app to check the label before sending the query to the local or remote model.
  4. The selected model processes the query, potentially using RAG, Google Search, or API calls for accurate responses.
  5. The response is shown to the user.

6. Example: Weather App

• The weather app answers questions about the weather.
• The router classifies queries based on whether they require local device information or external data.

7. Evaluation

• Evaluation is crucial for both the router and the LLM responses.
• Factors to consider:
  • Quality of responses.
  • Latency (response time).
  • Labeling outcome (accuracy of the router).
• The goal is to optimize the app for the specific use case.

8. Conclusion

• A hybrid LLM architecture allows you to combine the strengths of both on-device (Gemma) and cloud-based (Gemini) models.
• The routing architecture, using a text classifier like Mobile BERT, is a practical approach to distributing tasks.
• Careful evaluation is essential for optimizing the performance and accuracy of the hybrid system.