The future of Chrome Extensions with Gemini in your browser

Chrome Extensions and AI: A Deep Dive

Key Concepts:

• Chrome Extensions
• AI (Artificial Intelligence)
• Gemini Nano (on-device and in the cloud)
• Chrome's built-in AI APIs (Prompt API, Translation API, Audio Understanding, Image Understanding)
• Service Worker
• Context Menu API
• Side Panel API
• Content Scripts
• Accessibility
• Project Mariner
• Puppeteer
• Chrome DevTools Protocol

1. Introduction: The Rise of AI in Chrome Extensions

• Chrome extensions have been around for a long time and are a great way to customize the browser.
• There has been a massive increase in interest in Chrome extensions, especially those related to AI.
• The number of AI-related extensions has doubled in the past two years.
• 10% of all extensions installed from the Chrome Web Store use AI.
• This trend is expected to continue due to:
  • Models getting better at advanced tasks (complex reasoning, understanding images, audio, and video).
  • Chrome's new built-in AI APIs, which allow the use of Gemini Nano on the client.
• Sebastian Benz from the Chrome extensions developer relations team at Google discusses the possibilities of combining AI and Chrome extensions.

2. Chrome Extensions: Customization and Ease of Implementation

• Chrome extensions are easy to implement and distribute.
• Knowledge of HTML, JavaScript, and CSS is sufficient.
• Extensions can be built and shipped without requiring a server.
• Extensions can be distributed using the Chrome Web Store.

3. Chrome's Built-in Gemini Nano APIs

• Google I/O announced the launch of Chrome's built-in Gemini Nano APIs.
• These APIs cover use cases from prompting and writing to translating.
• AI can be used in extensions on the client without requiring an additional server or cloud service.
• The built-in prompt API will initially launch exclusively for Chrome extensions.
• Tom's talk about Chrome's new built-in AI APIs provides more information (link in description).

4. Calendar Mate: A Prompt API Example

• Functionality: Creates calendar events from text on a website.
• Process:
  1. User selects text on a website.
  2. Right-clicks and selects "Create Calendar Event."
  3. The extension extracts relevant event data from the text.
  4. Prefills a Google Calendar event.
• Implementation Details:
  • The extension service worker registers a new context menu entry called "CreateCalendarEvent."
  • Context menu actions are registered for specific content types (text).
  • A callback is registered for when the extension is invoked.
  • The selected text is accessed in the event handler and passed to the built-in prompt API to extract event data.
  • Temperature and top K are set to 1 for more predictable results.
  • Event details are added as parameters to the Google Calendar event creation page URL, which is then opened in a new tab.
• Key Advantages:
  • Use-case driven AI integration (no direct user interaction with the language model).
  • Tighter integration of an existing product (calendar) into the browser.
  • Avoids sending user data to a server for processing (privacy-focused).
• The Prompt API will be shipping in Chrome version 138.

5. Audio Scribe: Audio Understanding Example

• Problem: Difficulty in listening to voice messages during meetings.
• Solution: Audio Scribe transcribes voice messages in chat apps.
• Process:
  1. User clicks the extension icon.
  2. The side panel opens.
  3. The side panel shows a transcription of all voice messages received in the current chat.
• Implementation Details:
  • Uses the side panel API to show extension-specific content beside the main web content.
  • Uses content scripts to inject JavaScript or CSS into web pages.
  • Content scripts identify audio elements and pass their audio source to the prompt API for transcription.
• Key Advantage: Improves an already existing product (web-based chat apps) with built-in AI.
• Uses the audio understanding capabilities of Gemini Nano.

6. Alt Texter: Image Understanding Example

• Problem: Providing Alt text for images on social media to help visually impaired users.
• Solution: Alt Texter proposes Alt text for images.
• Process:
  1. User right-clicks any image on a web page.
  2. Selects "Generate Alt Text."
  3. A popup opens proposing a textual description of the image.
• Implementation Details:
  • Registers a content menu action for images.
  • Displays the generated Alt text in a popup.
  • Integrates the option to translate the generated Alt text using the new translation API.
• Key Advantage: Demonstrates how to use the new multimodal Gemini Nano APIs to make the web more accessible to users.

7. Project Mariner: A Complex Extension Example

• Description: A research prototype that takes action on your behalf in your browser.
• Functionality: Automates tasks in the browser.
• Example: Asking the agent to help improve Chrome extensions documentation by adding more code samples.
• Process:
  1. The agent comes up with a plan.
  2. Uses Google Search to find the Chrome extensions reference documentation.
  3. Picks the most fitting search result.
  4. Starts looking for API docs.
  5. Goes through all the APIs listed in the sidebar.
  6. Opens the respective documentation page.
  7. On an API page, it does a visual search for code snippets.
  8. Collects all pages that don't have a code sample.
• Implementation Details:
  • Uses the extension side panel API to display what the agent is currently doing.
  • Takes a screenshot of the page and uses the DOM to identify the position of possible actions.
  • Sends the screenshot with the annotated actions to a server that uses Gemini to decide what the next step should be.
  • The server sends the next action to perform, together with the explanation, back to the Chrome extension.
  • The Chrome extension then performs the action on the page, observes the new state, and sends it back to the server.
  • Uses Puppeteer to interact with web pages (take screenshots, analyze the DOM, perform actions).
• Key Advantage: Demonstrates what's possible with Chrome extensions by combining powerful extension APIs and the latest Gemini models.

8. Conclusion

• The APIs discussed are available to developers.
• Check out the extension documentation and the latest Gemini APIs.
• Join the extensions developer mailing list for questions.
• Samples mentioned in the video are linked in the description.