Build new features using built-in AI in Chrome

Key Concepts

• Built-in AI: AI models running locally on the user's device within the Chrome browser.
• Privacy-First AI: Data processing occurs on-device, ensuring sensitive information never leaves the browser.
• Hybrid Inference: A fallback mechanism using Firebase AI Logic to route requests to the cloud when on-device capabilities are unavailable.
• Structured Output: Using JSON schemas to force AI models to return data in a predictable, machine-readable format.
• Multimodal Input: The ability for AI models to process both text and images simultaneously.
• Polyfills: Code wrappers that allow developers to use the same API syntax across different platforms, even if native support is missing.

---

1. Built-in AI APIs Overview

Thomas Steiner introduced several Chrome-native APIs designed to enhance web applications without requiring server-side AI infrastructure:

• Summarizer API: Used for generating concise titles and SEO-friendly meta descriptions. It supports streaming responses for better performance.
• Prompt API: The core engine for general tasks. It supports Structured Output (via JSON schema) and Multimodal Input (processing images and text together).
• Writer & Rewriter APIs: Tools for content generation and refinement. The Writer API can expand bullet points into full drafts, while the Rewriter API adjusts tone, length, or format.
• Translator API: Enables real-time, on-device translation between languages.
• Language Detector API: A background utility that automatically identifies the input language for other APIs.

2. Technical Implementation & Frameworks

• Performance: By running locally, these APIs leverage hardware acceleration, often outperforming cloud-based solutions while maintaining offline functionality.
• Structured Output Methodology: To ensure the AI returns usable data (e.g., tags or moderation flags), developers define a response_constraint using a JSON schema. This forces the model to return a specific object structure, which is then parsed via JSON.parse().
• Hybrid Inference Strategy: For platforms lacking native support (e.g., mobile), developers can use the Firebase AI Logic hybrid SDK. This allows for "intelligent routing," where the app prefers on-device inference but falls back to the cloud if necessary.
• Polyfills: Julia (the developer in the story) created experimental polyfills for the Prompt and Task APIs, allowing developers to write code once and have it function across both desktop (native) and mobile (cloud-backed) environments.

3. Real-World Applications & Case Studies

• Drupal AI Team: Implemented the Summarizer API in their CKEditor for automated SEO tag and description generation.
• Cafe24: Uses the Prompt API with multimodal input to analyze product images and text simultaneously, turning simple descriptions into SEO-optimized content.
• CyberAgent (Ameba Blog): Utilizes the Writer and Rewriter APIs to suggest sentence completions and allow users to refine text tone.
• Policybazaar: Employs the Translator API to provide dynamic, location-based language support for their financial services.
• Yahoo Japan: Uses the Prompt API with structured output to classify user-generated questions for moderation, flagging harmful or illegal content.
• Plaid: Leverages on-device AI to act as a real-time mentor for travel agents, processing sensitive face-to-face conversations securely without cloud uploads.

4. Key Arguments

• Privacy as a Feature: Steiner emphasizes that because the AI runs locally, it is inherently more secure for sensitive data, a major selling point for enterprise and personal applications.
• Content-First Development: By using static site generators (like Eleventy/Build Awesome) combined with built-in AI, developers can achieve high performance (Lighthouse scores of 95+) while still providing a rich, dynamic editing experience.
• Developer Experience: The team provides TypeScript types (@types/dom-chromium-ai) and a starter template to lower the barrier to entry for developers wanting to integrate AI.

5. Notable Quotes

• "With the AI model running on the device, sensitive data never needs to leave the user's browser, which means huge benefits from a privacy perspective." — Thomas Steiner
• "For us, multimodal was not just an additional feature, it was the key result to quality." — Jerry Young Lee (Cafe24) regarding the integration of image and text processing.

6. Synthesis and Conclusion

The presentation demonstrates that the era of "Built-in AI" in Chrome is a reality, moving AI from a server-side luxury to a client-side standard. By utilizing APIs like the Summarizer, Prompt, and Translator, developers can build highly responsive, privacy-conscious, and offline-capable applications. The combination of Structured Output and Hybrid Inference provides a robust framework for production-grade apps, while the provided starter templates and polyfills ensure that developers can begin experimenting immediately.

Actionable Resources:

• Starter Template: goo.gle/starter-extended-blog
• Documentation: goo.gle/built-in-ai-docs
• Types: @types/dom-chromium-ai on NPM.