Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy

Key Concepts

• Hybrid RAG (Retrieval-Augmented Generation): A system combining semantic search (vector-based) and keyword search (BM25) to improve retrieval accuracy.
• Docling: A tool used to convert raw documents (PDFs, Word, images) into structured Markdown for cleaner data ingestion.
• Chunking Strategies: Methods to segment data, including Heading-based, Paragraph-based, Fixed-size (512 characters with 64-character overlap), and Sentence-based.
• Observability: Using LangFuse to track latency, chat history, and system performance.
• Guardrails: Implementing code-based logic (regex, dictionaries, and classifiers) to block prompt injections and irrelevant queries before they reach the LLM.
• Local LLM Stack: Utilizing Ollama with lightweight models (e.g., Qwen 2.5 0.5B) to ensure privacy, reduce costs, and maintain speed.

---

1. Main Topics and Technical Architecture

The presentation outlines a framework-free, hybrid RAG system designed for production environments. The core philosophy is to move away from "drag-and-drop" document uploading, which wastes tokens and creates messy data, toward a "structure-first" approach.

• Tech Stack: Python (FastAPI), React (Frontend), PostgreSQL (Vector Database), Docker (Containerization), Ollama (Local LLM execution), and LangFuse (Telemetry).
• Data Ingestion Pipeline: Raw documents are converted to Markdown via Docling, chunked using specific strategies, embedded, and stored in PostgreSQL.
• Hybrid Search: The system performs both semantic search (cosine distance) and keyword search (BM25) to ensure that specific terms (like SKUs or medical names) are retrieved alongside contextually relevant information.

2. Chunking Methodologies

The speaker emphasizes that the quality of RAG depends heavily on how data is prepared.

• Heading-based: Best for structured documents (e.g., handbooks). It maps chunks to specific headings, making it easy to trace the source of an answer.
• Paragraph-based: Useful for general text where headings are absent.
• Fixed-size (512 chars): A fallback for unorganized data, utilizing a 64-character overlap to maintain context between chunks.
• Sentence-based: Used for short, unstructured inputs like emails or screenshots.

3. Agent Mode vs. Direct RAG

• Direct RAG: A fixed, predictable pipeline. It is preferred for compliance-heavy tasks where hallucinations must be minimized.
• Agent Mode: Uses Python functions to perform tasks (e.g., fetching the current date or calculating prices). The speaker argues for using Python functions instead of LLM agents for these tasks to increase speed, reduce latency, and eliminate the risk of the LLM "making up" data.

4. Guardrails and Safety

To prevent prompt injection and irrelevant queries, the system implements "pre-LLM" filtering:

• Logic-based Blocking: Instead of relying on complex system prompts, the system uses hard-coded logic (regex and term dictionaries) to identify and block sensitive or out-of-scope queries (e.g., medical advice).
• Testing: Because the guardrails are code-based, they can be subjected to rigorous unit testing to ensure consistent behavior.

5. Notable Quotes

• "You don't need the biggest model. If you can vet your data before sending it to your LLM, a smaller model can sort you out and get a good answer for you."
• "The point of lots of these guardrails is that the issue should stop before we... send it to the LLM to generate anything."

6. Synthesis and Conclusion

The primary takeaway is that data quality and observability are more important than model size. By using a "structure-first" approach with Docling, implementing a hybrid search (Vector + BM25), and enforcing guardrails via code rather than prompts, developers can build highly accurate, low-latency, and cost-effective RAG systems. The system is designed to be fully local, ensuring data privacy and eliminating the need for expensive, proprietary frameworks.