Building a VR Assistant and llms.txt widget

Key Concepts

• Built-in AI in Browsers: The integration of Artificial Intelligence capabilities directly within web browsers.
• Google Chrome APIs: A set of JavaScript APIs available in the Chrome browser for various functionalities.
• Speech-to-Text (STT) API: Converts spoken language into written text.
• Text-to-Speech (TTS) API: Converts written text into spoken language.
• Translator AI API: Translates text from one language to another.
• Prompt API: An API for interacting with AI models, likely for generating text or performing tasks based on prompts.
• Response Constraint: A feature allowing AI models to return structured data, such as JSON.
• LLM.ext (Large Language Model Extension): A proposed web standard for using Large Language Models (LLMs) directly in the browser.
• Web Machine Learning (WebML): A proposal for enabling machine learning models to run directly in the browser.
• MCP (Machine-learning Compute Platform) Server: A concept where a website acts as a server exposing its backend as a public API for AIs to use.
• Firebase: A platform that can be used as a fallback for AI computations if the user's machine lacks the necessary resources.

Project 1: AI-Powered Barber Appointment Scheduling

Problem Statement

The speaker faced two main personal issues related to scheduling barber appointments:

1. Slow and Cumbersome Booking Process: The existing barber website required multiple steps, including logging in, selecting service categories, choosing specific services, selecting a professional, and then repeatedly checking for available slots, which was time-consuming (around 40 seconds per check).
2. Poor Personal Time Management: The speaker admitted to being "not so good with timing" and often forgot to book appointments in advance, leading to unavailability of preferred barbers.

Proposed Solution

To address these issues, the speaker envisioned a tool, potentially a Google Assistant task or a Chrome extension, that could:

• Check the barber's agenda automatically.
• Proactively inform the user about availability.
• Facilitate quick scheduling with user confirmation.

Technical Implementation and Methodology

1. Reverse Engineering the Barber's Application:

   • The speaker inspected the barber's website using browser developer tools to understand how requests were made.
   • They noted a lack of security measures, such as tokens, which made it easier to interact with the application's backend.
   • This allowed them to understand the logic for scheduling and message retrieval.

2. Leveraging Native Browser APIs (Chrome Desktop):

   • The project was built using only Google Chrome APIs, with the intention of future cross-browser compatibility.
   • Speech-to-Text API: Used to convert the user's spoken commands into text.
   • Translator AI API: Employed to translate user input from any supported browser language into English for processing. This ensured broad language support.
   • Prompt API: Used to send processed requests to an AI model.
   • Text-to-Speech API: Used to provide spoken responses to the user.

3. AI Interaction and Data Parsing:

   • Intent Parsing: The AI needed to understand the user's intention (e.g., booking, cancelling, inquiring about availability).
   • Entity Extraction: Key information like professional names, dates, and times were extracted from the user's request.
   • JSON Formatting: The extracted information was structured into a JSON object, including intention, professional_id, datetime, etc. This structured data was crucial for the AI to process and for the application to interact with the barber's system.
   • Response Constraint: The speaker highlighted the importance of the "response constraint" feature, which allows the AI to return structured data (like JSON) directly, simplifying the process of getting actionable information.
   • Prompt Engineering: The speaker provided directions and keywords to the AI model to guide its responses and ensure it adhered to the intended schema.

4. Workflow:

   • User speaks a command (e.g., "Book an appointment with Luchiano tomorrow at 11 AM").
   • Speech-to-Text API converts speech to text.
   • Translator AI API translates the text to English (if necessary).
   • The text is parsed to identify intent and entities, formatted into a JSON object.
   • The JSON object and a prompt are sent to the Prompt API.
   • The AI processes the request, potentially checking availability with the barber's system (simulated or actual).
   • The AI generates a response, potentially in JSON format.
   • The response is translated back to the user's language (if necessary).
   • Text-to-Speech API converts the response to speech.
   • The user hears the confirmation or relevant information.

5. Example Interaction:

   • User: "Hello. How you doing?"
   • AI: "I'm doing well. Thank you for asking. Is there anything I can help you with regarding your barber appointment today?"
   • User: "Which professionals I can choose?"
   • AI: "We have Luchiano, Shua, and Cohen. Would you like to book an appointment with one of them?"
   • User: "Is Luciano available tomorrow at 11:00 a.m.?"
   • AI: "Luchiano Syla is available at the requested time. Would you like to schedule now?"
   • User: "Yes, please."
   • AI: "Your appointment for Luchiano Syla at 11:00 a.m. is confirmed."

6. Cancellation Example:

   • User: "Cancel my appointment with Luchiano SA at 11:00 a.m."
   • AI: "Your appointment with Luchiano SA at 11:00 a.m. has been cancelled."

7. Technical Details:

   • The project ran on a Mac M1 with 16GB of RAM, demonstrating that powerful hardware is not always required for these browser-based AI tasks.
   • The speaker emphasized the ability to support "any language the browser is currently supporting" due to the Translator API.

Project 2: Offline, Private, In-Browser LLM Widget

Problem Statement

The increasing use of AI bots on websites leads to high CDN costs for companies as these bots scrape data. The speaker sought a more cost-effective and private solution.

Proposed Solution

To create a widget that can reply to users offline and privately within the browser, using a single text file containing all the website's information. This leverages the emerging LLM.ext web standard.

Technical Implementation and Methodology

1. LLM.ext Standard: The project utilizes the proposed LLM.ext web standard, which aims to enable Large Language Models to run directly in the browser.
2. Single Text File Data Source: All information about the company (courses, links, tracks) is consolidated into a single text file.
3. Offline and Private Operation: The LLM runs locally on the user's machine, meaning:
   • No Token Usage: No API calls to external services are made, saving costs.
   • No Third-Party Payment: Users do not incur charges for using the AI.
   • Privacy: Data remains on the user's device.
4. Real-World Application: The speaker mentioned a company in Brazil that teaches JavaScript developers, where this approach could be applied to provide course information and answer student queries without external dependencies.
5. Speed: The speaker highlighted the speed advantage of running AI locally, as it utilizes the "best machine of the world – the user's machine."

Key Arguments and Perspectives

• AI in the Browser is Transformative: The speaker strongly believes that built-in AI in browsers will fundamentally change how users interact with the web, eliminating the need for cumbersome multi-step processes and manual API hacking.
• Empowering Developers: New web standards like LLM.ext and WebML are empowering web developers to integrate sophisticated AI functionalities directly into their applications.
• Cost and Privacy Benefits of Local AI: Running AI models locally in the browser offers significant advantages in terms of cost reduction (no API fees, reduced CDN usage) and enhanced user privacy.
• Website as an MCP Server: The concept of websites acting as "MCP servers" by exposing their backend as public APIs for AIs to consume is presented as an exciting future direction.

Notable Quotes

• "I'm the only thing standing between you and your lunch. So, I promise I'm going to be quick." - Speaker, setting an energetic tone.
• "built-in AI, which for me it's going to change the whole world for us." - Speaker, expressing strong conviction about the impact of browser AI.
• "we are web developers. We don't want to wait for 40 seconds or so to browse, find a time or schedule an appointment." - Speaker, highlighting a common developer frustration.
• "we don't want we are not so good with timing, right?" - Speaker, humorously admitting a personal challenge addressed by the AI solution.
• "This is just running in the browser and it it didn't fry my computer. I didn't have a super computer. It was running on a Mac M1. 16 gigabytes of RAM. So my god this module is amazing." - Speaker, emphasizing the efficiency and accessibility of browser AI.
• "AIs were built about this right people say that AI was built on top of understanding how humans speak or think." - Speaker, connecting AI development to human cognition.
• "the whole goal here is to use your website as an MCP server." - Speaker, describing a future paradigm for web development and AI interaction.
• "We live in the future, right?" - Speaker, reflecting on the current advancements in web technology.

Step-by-Step Processes and Methodologies

• Barber Appointment Scheduling:

  1. User speaks a request.
  2. Speech-to-Text API converts speech to text.
  3. Translator API translates to English.
  4. Text is parsed for intent and entities.
  5. Data is structured into JSON.
  6. JSON and prompt sent to Prompt API.
  7. AI processes request and interacts with barber's system (simulated).
  8. AI generates a structured response.
  9. Response is translated back to user's language.
  10. Text-to-Speech API converts response to speech.

• Offline LLM Widget:

  1. Website data is compiled into a single text file.
  2. LLM.ext standard is implemented in the browser.
  3. LLM loads the text file locally.
  4. User interacts with the widget.
  5. LLM processes queries using local data and model.
  6. Replies are generated and displayed offline and privately.

Logical Connections Between Sections

The presentation flows logically from a personal problem to a technological solution, demonstrating the practical application of emerging AI technologies.

• The barber appointment problem serves as a relatable entry point, illustrating the inefficiencies of current web interactions.
• The technical implementation of the barber scheduler showcases how existing browser APIs (STT, TTS, Translator, Prompt) can be combined with AI to create intelligent agents.
• The discussion of data parsing and response constraints highlights the advancements in AI model interaction, enabling structured data exchange.
• The transition to the offline LLM widget project builds upon the theme of AI in the browser, focusing on a different use case (information retrieval) and emphasizing cost and privacy benefits.
• The introduction of LLM.ext and WebML positions these projects within the broader context of future web standards.
• The concept of an MCP server offers a forward-looking perspective on how websites and AI will interact.
• The next steps and conclusion tie everything together, reinforcing the transformative potential of these technologies and encouraging community involvement.

Data, Research Findings, or Statistics

• 40 seconds: The approximate time the speaker had to wait to find an available barber appointment slot on the website.
• Mac M1, 16GB RAM: The hardware specifications on which the AI-powered barber scheduler was successfully run, indicating modest resource requirements.
• "Any language the browser is currently supporting": The speaker's claim regarding the language support enabled by the Translator API.

Section Headings

• Introduction and Personal Motivation
• Project 1: AI-Powered Barber Appointment Scheduling
  • Problem Identification
  • Technical Approach and APIs Used
  • AI Interaction and Data Handling
  • Demonstration and Workflow
• Project 2: Offline, Private, In-Browser LLM Widget
  • Problem of AI Scraping Costs
  • Leveraging LLM.ext for Local AI
  • Benefits: Speed, Cost, Privacy
• The Future of Web AI: LLM.ext and MCP Servers
• Next Steps and Conclusion

Synthesis/Conclusion

The video effectively demonstrates the practical and transformative potential of integrating AI directly into web browsers. The speaker showcases two compelling projects: an AI assistant for streamlining personal tasks like barber appointments, and an offline LLM widget for efficient and private information retrieval. By leveraging native browser APIs and emerging standards like LLM.ext, these solutions offer significant improvements in user experience, cost-effectiveness, and privacy. The core message is that AI in the browser is not a distant future but a present reality that will reshape web development and user interaction, moving towards more intelligent, responsive, and personalized online experiences. The speaker encourages developers to explore these technologies and contribute to their advancement.