This Hybrid RAG Trick Makes Your AI Agents More Reliable (n8n)

Hybrid Search Implementation for AI Agents: Superbase and Pinecone

Key Concepts:

• Vector Search (Semantic Search): Captures the semantic intent of a query using dense vector embeddings.
• Keyword Search (Full Text Search): Matches exact or partial words using sparse vectors.
• Hybrid Search: Combines vector search and keyword search for improved accuracy and relevance.
• Dense Vectors: Numerical representations of text used in semantic search.
• Sparse Vectors: Representations that highlight specific keywords, used in keyword search.
• Reciprocal Rank Fusion: A process to merge and rank results from both vector and keyword search.
• Re-ranking: Using a model like Cohere to reorder the fused results for improved accuracy.
• Edge Functions: Serverless functions in Superbase that can be triggered by HTTP requests.
• Upsert: An operation that inserts new data or updates existing data in a vector store.

1. Introduction to Hybrid Search

The video addresses the problem of inaccurate results when using vector stores to ground AI agents, particularly when dealing with specific names, terms, acronyms, or codes. It proposes hybrid search as a solution, combining the strengths of both vector search (semantic understanding) and keyword search (precision).

2. Vector Search (Semantic Search) Explained

• Functionality: Vector search excels at capturing the semantic intent of a user's query.
  • Example: A query like "Can I see the various blue cotton t-shirts?" will retrieve not only blue t-shirts but also other cotton t-shirts or similar items.
• Process:
  1. The query is converted into a dense vector embedding.
  2. This vector is compared to existing vectors in the knowledge base.
  3. Similar results are retrieved based on vector similarity.
• Strengths: Captures the meaning and context of the query, returning a broad selection of relevant results.
• Weaknesses: Can be too broad and may struggle with specific terms or codes.
  • Example: A query like "Show me blue t-shirts that are medium-sized" might return results that are not medium-sized.

3. Keyword Search (Full Text Search) Explained

• Functionality: Keyword search matches exact or partial words in the knowledge base.
• Process:
  1. The query is converted into a sparse vector.
  2. The vector store returns results that contain the specific keywords.
• Strengths: High precision, ideal for finding exact matches.
  • Example: A query like "Do you have a black cotton t-shirt?" will return a product like "Apex 25 black cotton t-shirt."
• Weaknesses: Lacks semantic understanding and flexibility.
  • Example: "T-shirt" will not be recognized as equivalent to "T" unless explicitly defined.

4. Hybrid Search: Combining Vector and Keyword Search

• Functionality: Hybrid search merges the results from both vector search and keyword search.
• Process:
  1. The query is converted into both a dense vector (for semantic search) and a sparse vector (for keyword search).
  2. Two separate result sets are generated using the respective algorithms.
  3. The results are then ranked based on weighted scores from both systems.
• Benefits: Combines the precision of keyword search with the semantic understanding of vector search.
  • Example: A customer searching for "blue cotton t-shirt" will receive results with the exact match at the top, along with other similar t-shirts.
• Implementation: Requires custom work as N8N's built-in vector store tools only support semantic search.

5. Superbase Hybrid Search Implementation

• Database Setup:
  1. Create a documents table with columns for embeddings (dense vectors) and full text search (TS vector).
  2. Enable the vector extension in Superbase.
  3. Set the correct number of dimensions for the embedding model (e.g., 1536 for OpenAI's text-embedding-3-small).
  4. Create indexes for both the full text search and vector search columns for faster retrieval.
  5. Create a hybrid search database function that performs both vector and full text searches and fuses the results using reciprocal rank fusion.
• Edge Function:
  1. Create an edge function to generate embeddings for the user's query using OpenAI.
  2. Provide the Superbase project with an OpenAI API key.
• N8N Workflow:
  1. Use a chat trigger to capture the user's query.
  2. Use an HTTP request node to trigger the Superbase edge function, passing the query as a parameter.
  3. Load data into the vector store using a data ingestion pipeline (e.g., from Google Drive or web scraping).
  4. Add a metadata column (JSONB type) to the documents table.
  5. Use an AI agent to generate answers based on the retrieved results.
  6. Modify the hybrid search function to output the content, metadata, and rankings from both vector and keyword searches.
• Example Query: "What are the rules for the engine air intake?"
• Specific Term Example: "ISO16220" - Demonstrates how full text search can excel where vector search fails.
• General Question Example: "What is the impact of wind on an F1 car?" - Shows where vector search provides better results.
• Re-ranking: Implement a re-ranking system using a model like Cohere to improve the accuracy of the fused results.

6. Pinecone Hybrid Search Implementation

• Index Creation:
  1. Create a Pinecone index with the dense vector type.
  2. Set the number of dimensions according to the embedding model (e.g., 1024 for multilingual-e5-large).
  3. Choose product as the metric (required for hybrid search in Pinecone).
• Ingestion Flow:
  1. Download the document (e.g., F1 technical regulations).
  2. Extract the text from the PDF.
  3. Chunk the document using a custom JavaScript chunking script.
  4. Embed the chunks using Pinecone's API (both dense and sparse embeddings).
     • Use input_type="passage" for embedding chunks.
  5. Upsert the vectors to Pinecone using the vectors upsert endpoint.
• Inference Phase:
  1. Use an AI agent to trigger an N8N workflow.
  2. Generate dense and sparse embeddings for the query.
     • Use input_type="query" for embedding the query.
  3. Query Pinecone using the query endpoint, passing the dense and sparse vectors.
  4. Send the retrieved chunks back to the AI agent to generate the output.
• Example Query: "Can you explain the plank assembly rules?"

7. Conclusion

Hybrid search offers a significant improvement over traditional vector search by combining semantic understanding with keyword precision. The video provides detailed instructions on implementing hybrid search using both Superbase and Pinecone, including database setup, edge function creation, N8N workflow design, and data ingestion/inference processes. The examples and explanations highlight the benefits of hybrid search in various scenarios, demonstrating its ability to handle both general queries and specific term searches effectively.