Stop babysitting your agents... — Brandon Walsenuk, Unblocked

Key Concepts

• Context Engine: A system that aggregates static corporate knowledge (docs, code) and runtime data (Slack, PR history, Jira) to provide AI agents with the necessary information to perform tasks accurately.
• Naive RAG (Retrieval-Augmented Generation): A basic approach where agents search through a data store; often fails due to "satisfaction of search" (stopping at the first result) and lack of reasoning.
• MCP (Model Context Protocol): A standard for connecting AI agents to data sources and tools.
• Social Graph: A mapping of team interactions, code ownership, and expertise, used to prioritize relevant information for an agent.
• Token Optimization: The process of providing only the most relevant, high-quality context to an LLM to reduce costs and improve reasoning performance.
• Truthiness: The ability of a system to resolve conflicting information (e.g., code vs. a Slack conversation from a CTO).

---

1. The Problem: "Babysitting" Agents

Brandon argues that current AI agents are like brilliant but "context-blind" software engineers. Without a proper context engine, developers spend their time "babysitting" agents—correcting their code, pointing them to the right files, and fixing errors caused by incomplete information. The core issue is not a lack of intelligence in the LLM, but a lack of contextual understanding.

2. Three Myths of AI Context

• Myth 1: Naive RAG is sufficient. Simply dumping documentation into a vector store leads to "satisfaction of search," where the agent stops at the first piece of data it finds, even if it isn't the best or most accurate solution.
• Myth 2: Connecting enough MCPs is enough. While MCPs provide "pipes" to data, they do not provide the reasoning required to understand how to use that data within a specific organizational structure.
• Myth 3: Larger context windows solve everything. Even with massive context windows, agents struggle to reason over raw data without structured relationships, entities, or prioritization.

3. The Context Engine Framework

A robust context engine must move beyond static file systems. Its methodology includes:

1. Unified System Context: Ingesting data from all SaaS tools, codebases, and communication platforms (Slack/Teams).
2. Targeted Retrieval: Using a social graph to identify which codebases and patterns are relevant to the specific user or task.
3. Conflict Resolution: Settling discrepancies between documentation, code, and human communication (e.g., prioritizing a CTO’s comment in Slack over outdated code).
4. Data Governance: Respecting permissions so that agents only access data the user is authorized to see.
5. Token Optimization: Compressing the retrieved information into a highly relevant "research packet" before sending it to the agent.

4. Key Lessons Learned

• Optimize for Understanding, Not Just Access: Providing access to tools is useless if the agent doesn't understand the organization's specific patterns (e.g., factory patterns, internal libraries).
• Surface Conflicts: Do not hide conflicting information; the engine must identify and resolve these to prevent the agent from making incorrect assumptions.
• Avoid Caching Answers: Because organizational knowledge changes daily, caching "good answers" leads to stale, incorrect information.

5. Real-World Application: The Social Graph

Brandon demonstrated an open-source tool (to be released) that generates a Social Graph of an engineering organization.

• Functionality: It maps who works with whom, who reviews whose code, and who the "experts" are in specific modules.
• Application: When an agent is tasked with a feature, the engine uses this graph to pivot toward the most relevant experts and codebases, ensuring the generated plan aligns with the team's actual working patterns.

6. Notable Quotes

• "The gap is not intelligence at this point. It is context."
• "An agent should write code that feels like it was written by someone who’s been on your team for years."
• "If you cache a correct answer and then tomorrow someone asks the same question... you probably lied to them now because things probably changed."

7. Synthesis and Conclusion

The transition from "human as the context engine" to "automated context engine" is essential for scaling AI in engineering. By moving away from naive RAG and toward a system that understands organizational relationships, resolves conflicts, and respects governance, teams can move from "babysitting" agents to having them act as autonomous, high-performing team members. The ultimate goal is to provide agents with a "research packet" that allows them to plan and execute tasks with the same accuracy as a senior engineer.