How to Build a Multi‑User AI Chat App with Convex

Multi-User Dinner Coordination System with AI - Detailed Summary

Key Concepts:

• Convex: A real-time database and backend platform, sponsored the video, crucial for managing state and server-side logic.
• Anthropics Haiku (LLM): Large Language Model used for AI-powered restaurant suggestions and chat interactions.
• Google Places API & JavaScript Maps API: Used for searching restaurants and displaying them on a map.
• TanStack Start: UI framework powering the application's frontend.
• Agentic Coding: Utilizing LLMs as agents to perform actions and interact with tools.
• Tool Calls: LLM's ability to invoke external functions (tools) to access real-world data or perform actions.
• Real-time Database: Convex's database providing automatic synchronization of data across multiple clients.

1. Application Overview & Functionality

The presented application is a multi-user dinner coordination system. It allows multiple users (demonstrated with Jacksone and Joe) to collaboratively decide on a restaurant. The core features include:

• Real-time Chat: A synchronized chat interface where users can discuss options.
• Restaurant Search: Integration with the Google Places API to search for restaurants based on user queries.
• Map Visualization: Displaying restaurant locations on a map using the Google JavaScript Maps API.
• Shortlist & Voting: A voting mechanism allowing users to create a shortlist of potential restaurants and vote for their preferred choice. Currently, "Taste of Greek" has two votes.
• AI-Powered Suggestions: An AI agent (powered by Anthropics Haiku) responds to user prompts (initiated with "@ai") by suggesting restaurants and adding them to the shortlist.

2. Setup & Environment Configuration

Setting up the application involves the following steps:

1. Code Retrieval: Download the source code from the GitHub link provided in the video description.
2. Convex Initialization: Run convex init and convex dev to initialize Convex deployment variables.
3. API Key Acquisition: Obtain API keys for:
   • Google Places API: Required for restaurant search functionality.
   • Google JavaScript Maps API: Required for map display.
   • Anthropics API Key: Specifically using the Haiku model to minimize costs.
4. Environment Variable Configuration: Set the Anthropic API key and Google API key as environment variables within the Convex instance. This is crucial as Convex will execute the LLM calls and Google API requests.
5. UI Framework: The application is built using TanStack Start, primarily impacting the frontend UI.

3. Convex Configuration & Architecture

Convex plays a central role in the application's backend. Key aspects include:

• Data Storage: Convex stores the chat thread, the shortlist of restaurants, and user votes. The example shows a thread with "Taste of Greek" on the shortlist with two votes.
• Functions: Convex functions are server-side code that can be invoked from the client or other functions. Examples include:
  • getOrCreateThread: Retrieves or creates a chat thread.
  • listAllMessages: Retrieves all messages in the chat thread.
  • placesRequests: Functions related to interacting with the Google Places API (locked for server-side execution only).
• Security: Functions like placesRequests are locked, meaning they can only be executed on the server. This prevents malicious clients from directly accessing sensitive API keys or performing unauthorized actions.
• Real-time Updates: Convex's real-time database automatically synchronizes data across all connected clients, ensuring a consistent user experience.

4. Message Handling & Real-time Synchronization

The process of sending and receiving messages is as follows:

1. User Input: A user enters a message in the chat input field.
2. useChatInput & useAction: The useChatInput component utilizes useAction from convex React to invoke the api.chat.sendMessage function on the server.
3. sendMessage Function (Convex): The sendMessage function on the Convex backend checks if the message contains "@ai".
   • Regular Message: If no "@ai", the message is saved to the message list using saveMessage.
   • AI-Triggered Message: If "@ai" is present, the function triggers the AI agent.
4. useUIMessages & Real-time Updates: The useUIMessages hook subscribes to api.chat.listAllMessages and receives real-time updates whenever new messages are added. This ensures that all connected clients (e.g., Jacksone and Joe's browsers) are automatically updated with the latest messages. The streaming option is enabled for a responsive experience.

5. AI Integration & Tool Calls

The AI agent, powered by Anthropics Haiku, is invoked when a message contains "@ai". The process involves:

1. Agent Initialization: The dinnerAgent is initialized using the Vercel AI SDK (currently, Convex doesn't directly support TanStack AI).
2. System Prompt & Tools: The agent is configured with a system prompt and a set of tools:
   • searchRestaurants: Searches for restaurants using the Google Places API.
   • getRestaurantDetails: Retrieves details about a specific restaurant.
   • addToShortlist: Adds a restaurant to the shortlist.
   • removeFromShortlist: Removes a restaurant from the shortlist.
   • showOnMap: Displays a restaurant on the map (client-side tool).
   • highlightShortlistItem: Highlights a restaurant on the map (client-side tool).
3. Tool Call Choreography: The LLM identifies the need for a tool call and generates a special message indicating which tool to use and with what arguments.
4. Tool Execution: The Convex backend executes the requested tool (e.g., searchNearbyInternal which calls the Google Places API).
5. Result Handling: The tool's result is returned to the LLM, which formats a markdown message summarizing the findings and sends it back to the chat.

Example: When a user types "@ai Thai restaurants", the LLM uses the searchRestaurants tool to find Thai restaurants, receives the results, and then generates a message listing the options.

6. Shortlist Management & Voting

Adding restaurants to the shortlist and voting is handled as follows:

1. addToShortlist Tool: When the AI suggests a restaurant, it uses the addToShortlist tool.
2. Server-Side Mutation: The addToShortlist tool triggers a server-side mutation (addInternal) that inserts the restaurant into the shortlist database.
3. Real-time Updates: The useQuery hook in the shortlist component subscribes to changes in the shortlist database. Whenever a restaurant is added or removed, the UI automatically updates.
4. Voting: Users can vote on restaurants in the shortlist, which updates the vote count in the database and is reflected in real-time for all users.

7. Data & Statistics

• Two votes were cast for "Taste of Greek" during the demonstration.
• The application utilizes Anthropics Haiku to reduce costs associated with LLM usage.
• The application leverages Google APIs for location-based services.

Quote: "Technically, I gotta tell you, this project would've been a real pain in the butt without convex." - Demonstrates the value Convex provides to the project.

Conclusion:

This project demonstrates a powerful combination of technologies – Convex, Anthropics Haiku, and Google APIs – to create a collaborative and AI-enhanced dinner coordination system. The use of Convex's real-time database and serverless functions simplifies backend development and ensures a responsive user experience. The integration of an LLM agent adds intelligence and automation to the process, making it easier for users to find and agree on a restaurant. The emphasis on security through server-side function locking is a best practice for protecting API keys and sensitive data. The project provides a solid foundation for building more complex multi-user applications with AI capabilities.