We Cut 94% of AI Coding Tokens With a Local Code Index - Rajkumar Sakthivel, Tesco

Key Concepts

• Context Window/Tokens: The data (code, files, instructions) sent to an AI model.
• Input vs. Output Cost: The economic reality that ~90% of AI API costs stem from input tokens (context), while only ~10% come from output.
• Local Search Layer: A middleware architecture that indexes code locally to retrieve only relevant snippets rather than sending entire files.
• Hybrid Search: Combining semantic (meaning-based) search with keyword (exact-match) search to improve retrieval accuracy.
• Context Compression: Techniques to reduce the size of retrieved code (e.g., keeping only function signatures and descriptions).
• Shared Indexing: A unified knowledge base that allows multiple AI coding tools (Cursor, Copilot, etc.) to share context and memory.

---

1. The Problem: The "Context Tax"

Raj and Foss discovered that their AI coding bills skyrocketed not because of the AI's "thinking" process, but because of inefficient context management.

• The Data: A typical query sent 45,000 tokens, but only 5,000 were actually relevant.
• The Analogy: It is like ordering one pizza and paying for nine extra pizzas that you never eat.
• The Financial Breakdown: 90% of costs are in the input. Cutting output by 75% only saves ~8% of the total bill, whereas cutting input by 94% saves ~61% of the total bill.

2. Failed Methodologies

Before building their solution, the team tested three ineffective approaches:

1. Prompt Engineering: Trying to tell the AI to "only look at relevant code." Failure: The cost is incurred the moment the context is sent to the model.
2. Model Settings: Adjusting max tokens or temperature. Failure: These affect output, not the expensive input.
3. Output Compression: Instructing the AI to write shorter answers. Failure: Since output is only 10% of the cost, the savings were negligible.

3. The Solution: Local Search Layer

The team built a local, privacy-focused search layer that sits between the codebase and the AI.

The 5-Step Process:

1. Decomposition: Break code into logical units (functions, classes, methods) rather than random chunks.
2. Hybrid Search: Run semantic search (for concepts) and keyword search (for exact names) simultaneously.
3. Shrinking: Reduce code to function names and descriptions (e.g., a 50-line function becomes 5 lines).
4. Dependency Tracking: Map which functions call which, ensuring related code is retrieved together.
5. Scoring/Filtering: Apply a scoring formula to discard irrelevant context before it ever reaches the AI.

4. Key Arguments & Evidence

• Hybrid Search Superiority: Semantic search misses exact names (e.g., "authenticate" vs. "auth"), while keyword search misses related concepts (e.g., "login" vs. "sign in"). Together, they reduce the error rate from 1 in 4 to 1 in 10.
• Simple Formulas > Complex Models: The team found that a simple weighted formula (50% meaning, 30% keyword, 20% recency) outperformed AI-based judgment, which was too slow (adding 2–3 seconds of latency).
• Performance Metrics: Testing on the FastAPI project showed a reduction from 83k tokens per query to 4.9k (94% reduction) with 90% accuracy.

5. Notable Quotes

• "Fix the input. That's where your money goes."
• "Simple formula beats the complex model most of the time."
• "We argue about which model is best, Opus or Sonnet... but the models may be 30% of the cost, but the other 70% is what you feed it."

6. Limitations and Real-World Application

• Real-world Savings: While 94% is the theoretical maximum, real-world savings are lower because modern tools already have some optimization.
• Codebase Complexity: The tool performs best on modular codebases where files have single responsibilities. It struggles with "spaghetti" code where files contain many unrelated functions.
• Speed vs. Perfection: The team prioritized a small, fast model for indexing to ensure re-indexing takes under a second.

7. Synthesis: The Future of AI Tooling

The most significant takeaway is the need for Shared Memory. Currently, tools like Cursor and Copilot operate in silos, forcing the user to explain the same codebase repeatedly. By building a shared local index, the team enabled "memory"—once the codebase is explained to one tool, the context is available to all others.

Conclusion: The path to affordable AI development is not choosing a cheaper model, but optimizing the "context pipeline." By moving from a "send everything" approach to a "search-and-retrieve" architecture, developers can drastically reduce costs while maintaining high accuracy.