Benchmarking semantic code retrieval on Claude Code — Kuba Rogut, Turbopuffer

Key Concepts

• Semantic Code Retrieval: Using vector embeddings to search for code based on meaning rather than exact keyword matches.
• Agentic Search: The process of AI agents navigating file systems (e.g., using grep) to find relevant context.
• Turbo Puffer: A serverless full-text and vector search database designed for high-performance AI applications.
• Context Bench: A benchmark framework that evaluates how effectively an agent identifies the correct files, lines, and symbols required to solve a coding task.
• Precision vs. Recall: Metrics used to evaluate search effectiveness; Precision measures the relevance of retrieved files, while Recall measures the ability to find all necessary files.
• Cache Compute: The concept of pre-indexing and embedding code to save future compute cycles during agentic reasoning.

---

1. Main Topics and Key Points

The presentation explores the integration of semantic search into "Claude Code" (an agentic coding tool) to improve performance. While traditional agentic search relies on grep (keyword-based), the speaker argues that semantic search provides significant performance gains by acting as a "cache" for semantic meaning.

• Performance Gains: Citing Cursor’s implementation, the speaker notes a 24% relative improvement in answer accuracy for their "Composer" model and a 2.6% increase in code retention.
• The "Cache Compute" Thesis: By chunking, embedding, and indexing code upfront, agents can retrieve relevant context faster and with fewer tokens compared to repeatedly grepping through a file system.
• Tooling: The speaker introduced "Turbo Grep," a CLI tool that uses a tree-splitter library to parse code, embed it via the Voyage Code 3 model, and upload it to Turbo Puffer.

2. Real-World Applications and Case Studies

• Cursor: A primary case study. Cursor uses semantic search to index codebases, resulting in higher user satisfaction and better code accuracy. They utilize advanced techniques, such as injecting "fake comments" into code before embedding to bridge the gap between human queries and raw code.
• Claude Code: The speaker benchmarked Claude Code against three conditions:
  1. Raw Claude Code: Default behavior.
  2. Windowed Grep: Claude Code with a 50-line read limit.
  3. Windowed Grep + Semantic Search: The experimental setup using Turbo Puffer.

3. Benchmarking Methodology

The speaker utilized the Context Bench framework, which focuses on the process of solving a task rather than just the final output.

• Evaluation Criteria: Did the agent find the correct files, lines, and symbols?
• Findings:
  • Precision: Semantic search significantly reduced "wasted" file reads. Raw Claude Code had a 65% file precision (1 in 3 reads wasted), while the addition of semantic search improved this to 87% (1 in 8 reads wasted).
  • Recall: Raw Claude Code performed well in file recall due to its aggressive exploratory nature, but struggled with line-level precision.
  • Task Specificity: Semantic search excelled at finding "behavior-adjacent" files (files related by logic rather than keywords), whereas grep was superior for tracing specific imports or keyword-heavy tasks.

4. Key Arguments

• Tool Selection: No single tool is a silver bullet. Agents need a hybrid approach where they can choose between keyword search (for specific symbols) and semantic search (for conceptual understanding).
• The "Grepping" Limitation: While grep is zero-cost and effective for small, simple tasks, it fails to scale for complex, multi-file, or multi-modal (video/audio) data.
• The Role of Documentation: Semantic search performance is heavily dependent on the quality of inline documentation. Code with well-written comments above functions yields significantly better embedding results.

5. Notable Quotes

• "We think about how embeddings are cache compute... you have this cache of the semantic meaning so that when an agent wants to understand how metadata filtering works, it can simply query... and get the chunks it needs a lot faster."
• "The people that provide these easy tools to shrink down these billion-token context windows into the right million will win in the long term."

6. Synthesis and Conclusion

The integration of semantic search into agentic coding workflows is not merely about replacing keyword search, but about providing a more efficient way to navigate large codebases. While raw agentic search (like grep) is effective for simple tasks, semantic search provides a necessary "semantic cache" that improves precision and reduces token waste. The future of AI coding agents lies in hybrid systems that intelligently select the right retrieval tool—semantic or keyword—based on the specific requirements of the task at hand.