Pros and cons of on-device AI

Key Concepts

On-device AI / Edge AI: Running AI models locally on a device (phone, robot, etc.) instead of relying on remote servers.
Foundation Models: Large, pre-trained AI models often accessed via API.
Latency: The delay between a request and a response – reduced with on-device AI.
Context Size: The amount of text/data a model can process at once.
Parameters: The variables a model learns during training; fewer parameters generally mean a smaller model.
Ingress/Egress Costs: Costs associated with data entering and leaving a server/cloud environment.
Hybrid Approach: Utilizing both on-device and remote models within the same application.
Firebase AI Logic: A tool facilitating the switching between local and remote AI models.

Introduction to On-Device AI (Edge AI)

The discussion centers around the concept of on-device AI, also known as edge AI, and differentiates it from the server-based AI applications previously covered in the “Real Terms for AI” series. Traditionally, the series focused on applications where the core logic and AI processing occurred on a server, including interactions with foundation models. On-device AI, however, involves running AI models directly on the device where the application is being used – be it a phone, a robot, or another embedded system. The core definition used is that both the AI agent/application and the model it utilizes reside on the same physical device.

Advantages of On-Device AI

Several key advantages of employing on-device AI are highlighted:

Reduced Latency: Because the model is local, there’s no network round trip required, resulting in significantly faster response times.
Enhanced Privacy & Security: Data remains on the device, minimizing the risk of exposure during transmission to a remote server.
Cost Savings: Avoiding per-token charges for foundation model API calls, as well as eliminating ingress/egress and hosting costs associated with custom model endpoints, can lead to substantial cost reductions.

Limitations of On-Device AI

Despite the benefits, on-device AI isn’t a universal solution. Two primary limitations are discussed:

Model Size & Context Size: Current on-device models typically have fewer parameters and smaller context windows compared to their cloud-based counterparts. This restricts the complexity of prompts and tasks they can effectively handle. For example, a model might struggle with large images or complex summarization tasks requiring extensive context.
Package Download Size & Computational Capacity: Including a model within an application bundle increases its size, potentially causing download issues, especially with poor connectivity. Furthermore, older devices or software versions may lack the necessary processing power to support on-device AI applications.

The Hybrid Approach & Tools

The discussion acknowledges that a “one-size-fits-all” approach isn’t feasible. A hybrid approach is often necessary, leveraging local models when possible and falling back to remote models via API when local processing isn’t sufficient or supported. This strategy allows developers to maximize the benefits of on-device AI while maintaining functionality across a wider range of devices and use cases.

Tools like Firebase AI Logic are presented as solutions to simplify this hybrid implementation. These tools automate the switching between local and remote models, reducing the development effort required to manage this complexity. They allow developers to utilize both model types within a single application, adapting to the specific needs of each prompt or application component.

Device Variability & Rapid Development

The speakers emphasize the rapidly evolving landscape of on-device AI capabilities. Support for local models varies significantly between devices and operating systems. Developers are strongly advised to verify the specific capabilities of the target environments before implementing on-device AI features. The capabilities are changing so quickly that constant monitoring and adaptation are crucial.

Real-World Considerations & Use Case Dependency

The conversation consistently returns to the importance of the use case. While on-device AI offers compelling advantages, its suitability depends entirely on the specific application requirements. Thorough testing is essential to ensure that the chosen model delivers the desired quality and performance for the intended task.

Quote: “It always depends on the use case. And I want to be clear, often the models that you can use on a device are great and they will work just fine for your use case, but you’ll need to test them out to make sure that they work for your specific use case and that you’ll get the quality that you want.” – Speaker (Aza)

Conclusion

On-device AI represents a promising advancement in the field of artificial intelligence, offering benefits in terms of speed, privacy, and cost. However, it’s not a replacement for traditional server-based AI. A pragmatic approach, often involving a hybrid strategy and careful consideration of the specific use case, is crucial for successful implementation. The technology is still in its early stages, and ongoing development promises to expand its capabilities and accessibility.