Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

Key Concepts

• AI Harness: A software layer surrounding an AI agent that provides grounding, stability, and deterministic control, ensuring the agent operates reliably within a defined environment.
• Agent Loop: The iterative process where an agent receives a task, executes tool calls, and processes feedback until completion.
• Guardrails: Constraints (e.g., max iterations, max messages) that prevent agents from drifting, wasting tokens, or entering infinite loops.
• Deterministic Verification: The practice of using code-based checks to validate agent actions (e.g., checking if a button was actually clicked) rather than relying on the model's self-reporting.
• Context Management: Techniques for compacting or trimming conversation history to stay within model limits while maintaining essential information.

---

1. The Purpose of AI Harnesses

The speaker argues that because modern frontier models are "black boxes" and compute/token costs are significant, developers need a way to ensure reliability. A harness acts as an anchor, similar to how a mountain climber uses a harness to stay attached to a stable point. It allows developers to control the environment, manage costs, and ensure that agents perform tasks consistently regardless of the underlying model's non-deterministic nature.

2. Anatomy of an Agent Harness

A robust agent harness consists of several key components:

• Tool Registry: A collection of defined functions (e.g., file system access, bash commands, browser interactions) that the agent can invoke.
• Guardrails: Logic that enforces boundaries, such as max_steps or max_messages, to prevent runaway execution.
• Context Manager: A mechanism to compress or manage the agent's memory to prevent context window overflow.
• Agent Loop: The runtime environment that orchestrates the interaction between the model, the tools, and the harness logic.
• Verify Step: A deterministic check performed after an action to confirm the task was actually completed successfully (e.g., running a linter or checking a URL).

3. Step-by-Step Methodology: Building a Harness

The speaker demonstrated building a "poor man’s harness" for a browser-based agent tasked with upvoting a post on Hacker News:

1. Initial State: A basic agent loop using a browser session (Playwright) that frequently failed or "lied" about its success.
2. Adding Guardrails: Implementing iteration limits and context trimming to prevent infinite loops and token waste.
3. Implementing Verification: Adding a verify_successful_upvote function that inspects the trace of events to determine if the action truly occurred, rather than trusting the LLM's output.
4. Injecting Deterministic Logic: Creating a login_handler that detects when the agent hits a login page, injects credentials securely from the environment, and resumes the task. This moves the responsibility of "knowing how to log in" from the LLM's prompt to the harness's code.

4. Key Arguments and Perspectives

• Prompting vs. Engineering: The speaker emphasizes that "prompting harder" is often the wrong solution for agent failure. Instead, building a harness allows the developer to solve problems through deterministic code, which is more reliable than relying on the model's reasoning.
• Reliability over Intelligence: By using a harness, developers can achieve high-quality results even with older or cheaper models (e.g., GPT-3.5 Turbo), as the harness provides the necessary structure and safety.
• Enterprise Application: The speaker notes that IBM uses these concepts in their "Open RAG" project to provide enterprise-level security and data handling for sensitive, siloed information.

5. Notable Quotes

• "The agent harness is everything around the model that gives it grounding in reality."
• "Step one to solving a problem is admitting you have one." (Referring to the importance of the harness detecting and reporting failures accurately).
• "2025 was the year of agents. 2026 is the year of harnesses."

6. Future Outlook

The speaker envisions a future (2027) of "dynamic on-the-fly generated harnesses." In this scenario, an agent would be capable of self-assessing its task, generating its own harness to ensure safety and accuracy, executing the task, and then returning the result—effectively creating a self-aware, guardrailed environment for every unique request.

Synthesis

The core takeaway is that as AI agents move from simple chat interfaces to autonomous task execution, the "harness" becomes the most critical piece of infrastructure. By shifting logic from the LLM's prompt to a deterministic, code-based harness, developers can transform unreliable, non-deterministic models into stable, secure, and cost-effective agents capable of performing real-world tasks.