RAG is dead, right?? — Kuba Rogut, Turbopuffer

Key Concepts

• RAG (Retrieval-Augmented Generation): A framework for enhancing LLM responses by retrieving external data. The speaker argues that "simple" RAG (one-time vector search) is evolving into "Agentic Retrieval."
• Agentic Search: An iterative process where AI agents use tools (grep, semantic search, filters) to progressively reason over and retrieve context.
• Semantic Search: Using vector embeddings to find contextually relevant information rather than just keyword matching.
• Hybrid Retrieval: Combining vector search, full-text search (BM25), and metadata filtering to improve accuracy.
• Merkle Trees: A cryptographic hash tree structure used to efficiently identify changes in codebases to avoid redundant re-indexing.
• Cache Compute: The concept that embeddings and semantic indices act as a cache for expensive LLM reasoning, saving tokens and time.

---

1. The Evolution of RAG to Agentic Retrieval

The speaker addresses the common narrative that "RAG is dead," clarifying that while simple, one-shot vector search is insufficient for complex tasks, the underlying concept of retrieval is more critical than ever.

• The Shift: Retrieval is moving from a single-step process to an iterative, agent-driven loop.
• Agentic Workflow: Agents now perform "grep, read, assess, and repeat" cycles. They use tools to narrow down massive datasets into highly relevant context windows.
• Performance: Data from Cursor indicates that integrating semantic search into agentic workflows yields significant accuracy gains (up to 24% in specific models) and improves user retention.

2. Case Study: Cursor’s Indexing Methodology

Cursor, a prominent user of TurboPuffer, serves as a model for efficient agentic search.

• The Problem: Re-indexing large codebases for every session is computationally expensive and slow.
• The Solution: Cursor utilizes Merkle Trees to compare codebases across a team. If a codebase is similar to one already indexed, the system only re-chunks and re-embeds the specific files that have changed.
• Impact: This approach optimizes resource usage while maintaining high-quality semantic search capabilities, leading to a 2.6% increase in code retention and a 2.2% decrease in dissatisfied user requests.

3. Methodology: "Cache Compute" vs. Per-Session Discovery

The speaker contrasts two approaches to agentic reasoning:

• Per-Session Discovery (e.g., Cloud Code): The agent performs a "grep" through the file system every time a query is made. This is computationally expensive, repetitive, and consumes a high volume of tokens because the agent must "re-learn" the codebase every session.
• Indexed Retrieval (e.g., Cursor): An upfront cost is paid to parse and embed the codebase. At runtime, the agent performs lightweight queries against this index. This acts as "cache compute," where the initial indexing cost is amortized over many future queries, resulting in faster, cheaper, and more accurate agent performance.

4. Key Arguments and Perspectives

• Context Window Limitations: The speaker argues that simply increasing context window sizes (e.g., to a trillion tokens) is not the solution.
• The "Right Million" Principle: Citing Google’s Jeff Dean, the speaker emphasizes that the goal is not to process a trillion tokens at once, but to use stage retrieval to narrow down the data to the "right million" tokens that are actually relevant to the task.
• Retrieval is Multi-faceted: Effective retrieval is not just vector search; it includes full-text search (BM25), regex, globbing, and metadata filtering.

5. Notable Quotes

• Jeff Dean (Google): "You don't need a trillion [tokens] at once, you need the right million."
• Kuba (TurboPuffer): "Retrieval is no longer just this simple one-time call to a vector DB. It's becoming super iterative and these agents are really understanding what they're searching and searching to understand more."

6. Synthesis and Conclusion

The "death of RAG" is a misnomer; rather, it is the death of naive RAG. The industry is transitioning toward Agentic Retrieval, where systems use sophisticated, iterative search strategies to provide LLMs with precise context. By treating semantic indices as "cache compute" and utilizing efficient structures like Merkle Trees, developers can build agents that are not only more accurate but also significantly more cost-effective and performant. The future of search lies in the ability to intelligently narrow down vast amounts of data into the specific, actionable context required for the task at hand.