User Signal Dies at the Retrieval Boundary - Sonam Pankaj, StarlightSearch

Key Concepts

• Agentic Workflow: An LLM-based system capable of reasoning, tool invocation, and interacting with the environment.
• Retrieval Boundary: The conceptual limit where static retrieval methods fail to provide context that leads to successful task completion.
• Utility Score: A metric that weights semantic similarity by the historical usefulness of a memory in achieving a successful outcome.
• Agent RX (Runtime Experience): A layer that allows agents to learn from outcomes in real-time without requiring retraining or manual prompt engineering.
• Skills: Baked-in reasoning patterns derived from accumulated memories that update the agent’s behavior dynamically.
• Context Stuffing: The inefficient practice of overloading an LLM with excessive, often irrelevant, information.

---

1. The Problem: Why Agents Fail

Surim highlights that while AI agents are designed to reason and use tools, they often fail to achieve traction (85% failure rate according to Gartner/McKinsey). The core issues identified are:

• Static Retrieval: 73% of pipeline failures are attributed to retrieval, not generation. Current systems rely on embedding similarity, which ignores whether the retrieved information actually leads to a successful outcome.
• The "Signal Death" Gap: Observability stacks capture traces and tool calls, and evaluation suites judge pass/fail outcomes, but these two systems are disconnected. The "eval signal" dies in the dashboard and is not fed back into the agent’s context or reasoning process.
• Inefficient Memory: Current memory systems (e.g., LangChain, Mem0) focus on user preferences or conversation history rather than "reasoning-based" memory that helps the agent learn from past mistakes.

2. The Solution: Agent RX and Utility Scores

To bridge the gap between evaluation and action, Starling Search developed Agent RX.

• Utility-Based Retrieval: Instead of retrieving by keyword or simple semantic similarity, the system uses a Utility Score. This score ranks memories based on their historical contribution to successful task execution.
• Memory as Reasoning: The system treats memory as "reasoning" rather than just "facts." For example, a customer support agent shouldn't just remember a user's name; it should remember that "if a user asks for a refund, check the settlement status first to avoid double-paying."
• Dynamic Skill Updates: Once a memory has been validated through multiple successful outcomes, it is "baked" into a Skill. This allows the agent to update its behavior without the engineer needing to manually rewrite system prompts or redeploy the model.

3. Methodology and Framework

The process of improving agent performance follows this loop:

1. Execution: The agent attempts a task using current retrieval methods.
2. Evaluation: The system monitors the trajectory, tool calls, and final output.
3. Feedback Loop: If the agent fails (e.g., returns "no product found"), the system captures the failure.
4. Utility Adjustment: The system updates the utility score of the retrieved memories based on the failure or success.
5. Refinement: Future retrievals are re-ranked using these updated scores, and successful reasoning patterns are promoted to "Skills."

4. Benchmarks and Performance

The speaker presented data showing significant improvements using this memory-refined approach:

• Towel Bench (Policy Adherence): Performance improved from 66% to 76% with memory, and reached 80% when skills were fully integrated.
• Agentic Tasks: In benchmarks testing reasoning and multi-step workflows, the refined memory system achieved 61.3% accuracy, compared to 35.7% (baseline) and 58.2% (standard memory systems).

5. Notable Quotes

• "We have been optimizing for the wrong things. We made wrong answers appear faster and cheaper, but we forgot to make retrieval learn." — Quoting Ram Sriram (ex-CTO of Pinecone).
• "The eval signal dies in the dashboard. This is a missing layer, a system that consumes traces, absorbs evals, and converts both into retrieval guidance for future runs." — Surim.

6. Limitations and Challenges

• Cold Start: The system relies on semantic search until enough outcomes are accumulated to build a utility history.
• Utility Drift: Similar memories may compete, requiring careful management of the "lambda" hyperparameter used in re-ranking.
• Noisy Labels: If the feedback/evaluation process is flawed, the utility scores can become inaccurate.

7. Synthesis

The core takeaway is that for AI agents to move beyond static Q&A and into reliable production workflows, they must transition from static retrieval to outcome-informed learning. By treating memory as a dynamic reasoning asset—weighted by utility and updated through real-world feedback—developers can create self-improving systems that evolve without the constant overhead of manual prompt engineering or model retraining.