Small language models with Google AI Edge

Key Concepts

• AI Edge: Google's suite of tools for running AI models on devices (mobile, web).
• LiteRT: Google AI Edge's runtime, designed for performance and ease of use.
• LLM Inference API: A simple API for running small language models on the edge.
• Model Customization: Fine-tuning, quantization, and architectural changes to models.
• Quantization: Reducing the precision of model weights (e.g., FP32 to Int4) to reduce size and improve latency.
• AI Edge Quantizer: Tool for quantizing models, now supporting Int4 post-training quantization.
• AI Edge Torch Generative API: API for building custom language model architectures using PyTorch and AI Edge Torch layers.
• Model Explorer: Graph visualization tool for inspecting ML models.
• AI Edge Portal: GCP service for testing models on real devices in a device lab.
• Retrieval Augmented Generation (RAG): Augmenting language models with external knowledge during inference.
• Function Calling (Tool Use): Enabling language models to trigger actions by executing APIs.
• GPU Delegate (ML Drift): Component for GPU acceleration, re-architected for better GenAI performance.
• NPU: Neural Processing Unit, specialized hardware for AI acceleration on mobile devices.
• AI Packs: Bundles containing compiled models, runtimes, and dependencies for different hardware vendors.

Benefits of Running AI on the Edge

• Real-time, low-latency processing: Critical for applications like video conferencing background removal.
• Offline use: Enables functionality without network connectivity (e.g., voice timers).
• Privacy advantages: Keeps user data on the device (e.g., voice message de-noising).
• Reduced server-side costs: Eliminates the need for server maintenance, capacity, and inference costs.

Google AI Edge Tools

• High-level APIs: For common tasks like image segmentation.
• Runtime interface: For implementing custom AI features.

Language Model Support and Customization

• LLM Inference API: Provides a simple prompt-in, response-out API.
• Cross-platform support: Web, iOS, and Android.
• LiteRT Community on Hugging Face: Sharing converted models ready for edge use, including DeepSeek and Gemma.
• Google AI Edge GitHub: Contains edge-optimized Python implementations of popular models.
• Fine-tuning: Using tools like Torchtune or Hugging Face Transformers to train models on custom data.
• Colab: A Colab is published which uses Hugging Face Transformers and Google AI Edge to run through every step you'll need to process from fine tuning to on-device model inference.
• AI Edge Quantizer: Supports Int4 post-training quantization, reducing model size by up to 8x.
• AI Edge Torch Generative API: Allows building custom model architectures with PyTorch and AI Edge Torch layers.

Inspectability and Testability

• Model Explorer: Graph visualization tool for understanding model structure, comparing variants, and inspecting quantization schemes. Has seen 100,000 downloads and growing.
• AI Edge Portal: GCP service for running models on real devices in a device lab, enabling benchmarks, bulk inference, and evals. Currently in private preview.

On-Device Retrieval Augmented Generation (RAG)

• RAG SDK: Enables feeding external knowledge to language models during inference.
• Components: On-device embedding model, chunking method, vector database, and retrieval function.
• Modular: Allows replacing individual components with custom implementations.
• Availability: Android (currently), iOS and web (later this year).

On-Device Function Calling (Tool Use)

• Function Calling SDK: Allows language models to trigger actions by executing APIs.
• Function Registration: Registering application functions with the SDK.
• Customization: Fine-tuning models for specific functions using synthetic data generation.
• Availability: Android (currently), iOS and web (later this year).

Infrastructure Improvements (LiteRT)

• GPU-first design: Runtime stack optimized for GPU acceleration.
• Simplified GPU usage: Reduced code required to leverage GPU acceleration (up to 80% less code).
• ML Drift: Re-architected GPU delegate for better GenAI performance (12x improvement compared to other frameworks).
• NPU support: Simplified process for leveraging NPUs, offering significant performance gains (25x vs. CPU, 8x vs. GPU).
• Vendor partnerships: Collaborations with Qualcomm and MediaTek for streamlined developer experience.
• AI Packs: Bundles containing compiled models, runtimes, and dependencies for different hardware vendors, distributed through Google Play Store.

Real-World Examples

• Ensign InfoSecurity: Reduced deepfake detection time from 67 seconds to 1.2 seconds using LiteRT quantization and CPU/GPU delegates.
• Argmax: Achieved a 30% performance improvement by switching from ExecuTorch to LiteRT and migrating to ML Drift.
• AudioShake: Enables real-time audio separation use cases with LiteRT's performance.

Conclusion

Google AI Edge is providing a comprehensive suite of tools and infrastructure to enable developers to bring powerful AI capabilities, particularly language models, to mobile and web devices. Key advancements include expanded model support, customization options (fine-tuning, quantization), RAG and function calling SDKs, and significant performance improvements through LiteRT's GPU and NPU support. These advancements are making on-device AI more accessible, performant, and practical for a wide range of applications.