Client side Web AI agents to build smarter future user experiences

Key Concepts

• Agentic Behavior: Autonomous task completion by breaking down tasks into structured steps without constant user intervention.
• Tools: Functions, APIs, or data sources that an agent can use to interact with the outside world and gather information.
• LLMs (Large Language Models): Models used for reasoning and generating responses within the agent.
• Orchestration Layer: The component that manages and coordinates the LLMs, memory, and tools within an agentic system.
• Web AI: Running machine learning models client-side in a web browser using JavaScript and related web technologies.
• AI Agent Compatibility: Designing websites to be easily used by AI agents for task completion.

Agentic Systems: The Core Components

An agentic system is comprised of:

• One or more Large Language Models (LLMs): These models provide the reasoning and decision-making capabilities for the agent. Multiple LLMs can be used, with an orchestration layer deciding which one to use for a specific subtask.
• Memory Implementation: This allows the agent to retain context from previous interactions and steps, enabling it to make informed decisions in subsequent steps.
• One or more Tools: These are functions, APIs, or data sources that the agent can use to interact with the outside world. Tools bridge the gap between the LLM's knowledge and real-time information or external actions.

These components are managed by an agent runtime or orchestration layer, which handles:

• Taking in external inputs.
• Looping for internal reasoning.
• Calling the correct tools based on structured outputs from the LLM.
• Asking follow-up questions to the user.
• Defining valid stopping points to present results.

The Role of Tools

Tools are crucial for enabling agents to perform real-world tasks. They allow the LLM to access information and perform actions beyond its inherent knowledge.

• Example: Instead of hallucinating the weather in London, an agent can use a tool to fetch the current weather from a weather API.
• Types of Tools:
  • Calling a function to perform a task.
  • Using an extension to access an API.
  • Fetching contextual information from a vector data store.

Web AI and Agentic Behaviors: A Prototype Demonstration

The video demonstrates a Web AI prototype that integrates agentic behaviors into a flight search website, similar to Google Flights. The prototype uses Google's 2 billion parameter Gemma 2 model, running entirely locally in the browser using the MediaPipe Web LLM library.

• Hardware: The demo runs in real-time on an NVIDIA 1070 GPU.
• Functionality: The agent can understand natural language requests, ask follow-up questions, extract relevant information, and use a search tool (simulated in the demo) to find flights.
• Example Interaction: The user asks to book a flight for two people but doesn't provide enough information. The agent asks for the departure location and date. The user then provides more information, including a request to pick a skiing destination in the French Alps. The agent suggests Chamonix and calls the search tool.
• Key Features Demonstrated:
  • Natural Language Understanding: The agent can understand complex requests and extract relevant information.
  • Follow-up Questions: The agent can ask for clarification when needed.
  • Tool Usage: The agent can use tools to access information and perform actions.
  • Local Execution: The entire process runs locally in the browser, without sending data to the cloud.
  • Adaptability: The agent can adapt to changes in plans, such as adding a passenger or changing the destination.

Caching Proxy Library

The presenter created a caching proxy library to fetch large generative AI models from a server.

• Functionality: The library fetches large binary files, shards them into smaller chunks to fit within client-side caching API limits, and reconstructs the binary blob as a data URL.
• Benefits: Simplifies the process of loading large models in the browser and improves performance by caching the model for future use.

Dataflow and Implementation Details

The agent orchestration logic attaches to existing business logic, allowing developers to expose existing website functions as tools for the agent.

• Dataflow:
  1. The web app downloads the LLM from Cloud Storage.
  2. The MediaPipe Web LLM library loads the model and runs it locally.
  3. Existing functions in the business application are registered as tools.
  4. The user speaks to the agent using a speech input layer.
  5. The speech is transcribed and routed to the agent controller.
  6. The LLM generates a response in a structured JSON format.
  7. The agent orchestration logic interprets the JSON and calls the appropriate tool.
  8. The results are rendered to the user interface.
• Agent Configuration: The agent's persona and available tools are defined in the side panel, using JSDoc comments.
• JSON Structure: The desired JSON structure for the LLM's output is specified in the agent's configuration, allowing the orchestration layer to route requests to the correct tools.

Web AI DJ Example

The presenter created a Web AI DJ in one day by reusing the framework from the flight search demo. This demonstrates the versatility of the approach.

• Functionality: The Web AI DJ uses public APIs for music services to play music based on user requests.
• Enhancements: The text-to-speech engine was upgraded to use a Web AI model called QoQoro for a more natural-sounding voice.

Web AI vs. Cloud AI

• Web AI: Runs models client-side in the browser, using JavaScript and web technologies.
• Cloud AI: Runs models on the server-side and is accessed via APIs.

The Future of Web AI

• Hardware Evolution: As hardware continues to improve, more powerful models will be ported to run in the browser.
• Hybrid Approaches: Combining Web AI and Cloud AI to leverage the strengths of both ecosystems. For example, using a local Web AI agent with a cloud-based vector database for domain-specific knowledge.
• AI Agent Compatibility: Websites will need to be designed to be easily used by AI agents.
• Publicly Accessible Agent Files: Websites could have a publicly accessible file that describes the tools and services they offer, allowing agents to easily understand and use them.

Conclusion

Web AI and agentic behaviors have the potential to revolutionize the way users interact with the web. By running models locally in the browser and designing websites to be AI agent compatible, developers can create more natural, efficient, and accessible user experiences. The presenter encourages developers to start exploring Web AI today and shape the future of the web.