Real-Time AI Voice Chatbot in Python

Key Concepts

• Real-time AI Voice Chatbot: A system that allows for natural, immediate voice interaction with an AI.
• Moshi: A speech-to-speech foundation model that handles both audio and text processing.
• Mimi: A neural audio codec integrated within Moshi, responsible for encoding audio into tokens and decoding tokens back into audio.
• Modal.com: An AI infrastructure platform used for deploying and scaling AI applications, particularly those requiring GPU resources.
• FastAPI: A modern, fast (high-performance) web framework for building APIs with Python.
• React: A JavaScript library for building user interfaces.
• WebSockets: A communication protocol providing full-duplex communication channels over a single TCP connection, enabling real-time interaction between client and server.
• Opus Codec: An audio codec used for efficient audio compression and transmission.
• PCM (Pulse Code Modulation): Raw, uncompressed audio data.
• GPU Instances: Dedicated hardware accelerators (Graphics Processing Units) used for computationally intensive tasks like AI model inference.
• VRAM (Video Random Access Memory): Memory on a GPU, crucial for loading and running large AI models.

Project Architecture and Components

The project aims to build a real-time AI voice chatbot by integrating several components:

1. Moshi Model: The core AI model responsible for speech-to-speech and speech-to-text functionalities. It leverages Mimi for audio encoding/decoding and a text tokenizer for text processing.
2. Modal.com Backend: Hosts and manages the Moshi model, providing on-demand GPU instances for inference. It uses FastAPI to create an API for communication.
3. React Frontend: A user interface built with React that handles user input (microphone) and displays AI output (text and synthesized speech).
4. WebSockets: Facilitate real-time, bidirectional communication between the React frontend and the FastAPI backend.
5. Opus Codec: Used for efficient handling of audio streams between the browser and the backend.

Moshi Model and Mimi Audio Codec

• Moshi's Functionality: Moshi is a speech-to-speech foundation model that takes acoustic audio data as input and produces both audio data (talking back) and text data (transcription).
• Mimi's Role: Mimi, integrated within Moshi, is a neural audio codec. It encodes raw audio into discrete "audio tokens" and decodes these tokens back into audible sound. This process is visualized as: Audio Data -> Tokens -> Mimi -> Audio Tokens -> Moshi -> Output -> Mimi -> Audio.
• Model Requirements: The Moshi model requires approximately 24 GB of VRAM, necessitating the use of powerful GPUs for its execution.

Deployment with Modal.com

• On-Demand GPU Instances: Modal.com is used to deploy the application and spin up GPU instances as needed. This approach ensures isolated environments and sufficient resources for multiple users, especially for a potential SaaS application.
• Ease of Use: Modal simplifies the process of booting GPU instances and running Python code with just a few commands.
• Free Credits: New users receive $5 in free credits upon signing up (no credit card required) and an additional $30 if a credit card is provided. This allows users to follow along with the tutorial for free.
• GitHub Repository: A pre-built example of the real-time chatbot is available on the Modal Labs GitHub repository, allowing users to deploy it directly.

Project Structure and Setup

The project is structured into a src directory containing Python files and a frontend directory for the React application.

• common.py: Defines the Modal application instance (modal.app).
• app.py: Serves the React frontend using FastAPI. It configures a Debian slim Docker image with Python 3.11, installs FastAPI, and statically serves the frontend assets. It also sets up CORS middleware to allow cross-origin requests.
• moshipy.py: Contains the backend logic, including WebSocket handling and model inference.

Development Environment Setup:

1. Account Creation: Create an account on Modal.com.
2. Virtual Environment: Set up a Python virtual environment (e.g., using uv init).
3. Install Modal: Install the Modal package (uv add modal or pip install modal).
4. Modal Setup: Run uv run modal setup or python3 -m modal setup to connect to your Modal account and authorize the API token.

Backend Logic (moshipy.py)

This file handles the core AI model interaction and WebSocket communication.

• Image Configuration:
  • Uses a modal.Image.debian_slim with Python 3.11.
  • Installs necessary Python packages: moshipy, phon (for Opus codec), fastapi, and huggingface_hub.
  • Configures a volume for model caching (/models) to avoid re-downloading models.
  • Defines image imports for PyTorch, NumPy, phon, sentencepiece, and Hugging Face Hub utilities.
• Modal Class (Moshi):
  • Annotated with app.cls to define a Modal class.
  • Configured with an A10G GPU (24GB VRAM) and a timeout of 600 seconds.
  • Maps the model cache volume.
• setup Method (@modal.enter):
  • Runs once when a Modal instance is created.
  • Sets the device to CUDA if available, otherwise CPU.
  • Loads Mimi weights and the Mimi model using hfhub_download and loaders.get_mimi.
  • Sets num_codebooks for Mimi to 8 (hierarchical encoding).
  • Calculates frame_size based on sample rate and frame rate.
  • Loads Moshi weights and the Moshi LM model.
  • Initializes an lm_gen instance for inference with a temperature of 0.8 and top_k=250.
  • Enables streaming for Mimi and lm_gen.
  • Loads the text tokenizer using sentencepiece.
  • GPU Warm-up: Performs dummy inference on four chunks of audio data to pre-load the GPU and reduce latency for the first user request. This involves encoding chunks with Mimi, generating tokens with lm_gen, decoding output tokens back to audio, and synchronizing CUDA.
• Web Application (@modal.asgi_app):
  • Defines an asynchronous FastAPI application named web.
  • WebSocket Endpoint (/ws):
    • Accepts WebSocket connections.
    • Resets Mimi and lm_gen for each new connection.
    • Initializes Opus stream readers and writers for audio data.
    • receive Function: Continuously receives bytes from the WebSocket, appends them to the Opus stream reader.
    • process Function:
      • Reads audio data from the Opus stream reader in PCM format.
      • Accumulates PCM data into a buffer.
      • When enough data is buffered, it processes chunks:
        • Encodes the chunk using self.mimi.encode.
        • Generates output tokens using self.lm_gen.step.
        • Decodes output tokens back into audio using self.mimi.decode.
        • Extracts text tokens from the output tokens.
        • Converts text tokens to human-readable text, handling special tokens (padding, separator) and replacing a specific Unicode character (U+2581) representing whitespace.
        • Sends data back to the frontend via WebSocket, prepending \x02 for text and \x01 for audio.
    • send Function: Continuously reads audio data from the Opus stream writer and sends it to the frontend via WebSocket, prepending \x01.
    • Runs receive, process, and send functions concurrently using asyncio.gather.
    • Handles exceptions by printing errors and cancelling tasks.
  • Returns the FastAPI web application.

Frontend Development (React)

The frontend is built using React and served statically.

• Directory Structure: frontend/index.html and frontend/app.jsx.
• index.html:
  • Includes necessary JavaScript libraries via CDNs: Tailwind CSS for styling, React, ReactDOM, and Opus-related packages for microphone and audio handling.
  • Includes the app.jsx file.
  • Defines a div with id="root" where the React application will be mounted.
• app.jsx:
  • Uses React hooks: useRef, useEffect, useState.
  • getWebsocketUrl Function: Dynamically constructs the WebSocket URL based on the current protocol (HTTP/HTTPS).
  • Main Application Component:
    • Manages state for displayed text (text, setText).
    • Uses refs for socket, decoder, and scheduledEnd to persist across re-renders.
    • startRecording Function:
      • Initializes the microphone and audio recorder.
      • Gets user media.
      • Sets the socket to ready state.
      • Opens the WebSocket connection.
      • Starts sending audio data.
    • useEfffect for Decoder Loading: Loads the Opus decoder from the embedded JavaScript package.
    • useEfffect for WebSocket Handling:
      • On WebSocket open, calls startRecording.
      • On message received:
        • If the message type is 1 (audio), plays the audio.
        • If the message type is 2 (text), updates the displayed text.
    • Audio Playback: Handles receiving audio data, creating an audio buffer, connecting it to the source, playing it, and calculating the scheduled end time.
    • Rendering: Displays either the generated text from the model or a "Connecting..." message during initial setup.
  • Root Rendering: Uses ReactDOM.createRoot to render the App component into the root element.

Running the Application

1. Deploy: Navigate to the src directory in your terminal and run: uv run modal serve src/app.py (or python3 -m modal serve src/app.py)
2. Access: Modal will provide a URL to access the deployed frontend in your browser.

Real-time Interaction and Performance

• The chatbot demonstrates real-time voice interaction, starting to respond as soon as the user stops speaking, and sometimes even mid-sentence.
• The initial response from the model might have some latency due to GPU warm-up, but subsequent interactions are faster.
• The example provided is a minimal implementation, and the speaker notes that more intelligence and functionality can be added.

Conclusion and Key Takeaways

The project successfully demonstrates the creation of a real-time AI voice chatbot by integrating a powerful speech-to-speech model (Moshi) with a scalable deployment platform (Modal.com) and a modern web stack (FastAPI and React). The use of WebSockets enables seamless, low-latency communication between the frontend and backend. Modal.com significantly simplifies the deployment and scaling of GPU-intensive AI models, making advanced AI applications accessible to developers. The free credits offered by Modal allow for experimentation and learning without upfront costs. The speaker emphasizes the ease of use of Modal and the impressive real-time capabilities achieved.