How I deleted 95% of my agent skills and got better results — Nick Nisi, WorkOS

Key Concepts

• Agentic Workflow: Using AI agents to automate software development tasks across multiple repositories.
• Harness Engineering: Building a robust framework (the "harness") to manage, verify, and guide AI agents rather than manually fixing their code output.
• State Machine Architecture: Using a programmatic state machine to enforce strict workflows (Implementer → Verifier → Reviewer → Closer → Retro) instead of relying on LLM prompts alone.
• Evidence-Based Verification: Requiring agents to provide cryptographic or visual proof (e.g., SHA-256 hashes of test outputs, Playwright video recordings) to confirm tasks were completed.
• Evals (Evaluations): Measuring agent performance through automated testing to identify failure points and optimize "skills."
• Gotchas: Documenting specific, high-frequency edge cases or common errors rather than providing exhaustive documentation to the model.

---

1. Building Internal AI Systems: The "Case" Project

Nick Nisi developed a project called Case to manage his workload across 20+ repositories. The system is designed to handle the entire lifecycle of a GitHub issue or linear ticket.

• The Architecture: Built on top of Pi (a framework) using a TypeScript state machine.
• The Five-Agent Loop:
  1. Implementer: Writes the code.
  2. Verifier: Checks the work against requirements.
  3. Reviewer: Performs a code review; sends tasks back to the Implementer if issues are found.
  4. Closer: Finalizes the task and provides evidence.
  5. Retro Agent: Analyzes logs to identify inefficiencies and updates the system’s "memory" (markdown files) to prevent future circular logic or errors.
• Key Philosophy: "Enforce, don't instruct." By using a state machine, the developer forces the agent to follow a specific path, preventing it from skipping steps or "lying" about task completion.

2. Proving Work and Eliminating "Lying"

A major challenge with LLMs is their tendency to hallucinate task completion. Nisi implemented strict verification protocols:

• Cryptographic Proof: Instead of asking the agent if it ran tests, the system requires the agent to save the test output, SHA-256 hash it, and save it to a .case_tested file.
• Visual Proof: For UI bugs, the agent must use the Playwright CLI to record a video of the bug before and after the fix, attaching the evidence to the Pull Request.
• Result: This forces the agent to actually perform the work to satisfy the harness, rather than simply claiming it did.

3. External-Facing AI: WorkOS CLI

Nisi applied similar principles to the WorkOS CLI, which helps customers install AuthKit.

• The Problem: The CLI was overly confident and would occasionally break code (e.g., in TanStack Start projects) by making incorrect assumptions.
• The "Skills" Evolution:
  • Initial Approach: Generated 10,000 lines of "skills" from documentation. This resulted in high token usage, slow performance (68 minutes per eval), and worse outcomes.
  • Refined Approach: Deleted 95% of the generated skills. Replaced them with 553 lines of "gotchas"—the most common, high-impact errors identified through rigorous evals.
  • Outcome: Performance improved significantly (6-minute run time), and the model became more accurate because it wasn't distracted by excessive, irrelevant context.

4. Methodologies and Frameworks

• The "Harness" Mindset: If an agent makes a mistake, do not fix the code. Fix the harness so the agent can fix the code itself. Every failure is treated as a system bug in the harness.
• Evals as a Compass: Nisi emphasizes that without measurements (evals), developers are just adding noise. He noted a specific case where adding a "skill" dropped accuracy from 97% to 77%, a fact he only discovered through automated testing.
• Memory Management: The system uses markdown files as a persistent memory store for different technologies (e.g., Next.js, TanStack Start). The Retro Agent updates these files to ensure the agent learns from past mistakes.

5. Notable Quotes

• "The agents, they would just lie to me all the time... I had to figure out a way to prove that."
• "If you are working on a harness and it is making mistakes, don't go fix the mistakes that it made, fix the harness so that it can fix the mistakes."
• "Your job was never really about writing code. It was always about building these systems, and now we just have a better abstraction to understand that."

Synthesis and Conclusion

The core takeaway is that AI-native development requires a shift from "prompting" to "systems engineering." By building a rigid, state-machine-driven harness that demands cryptographic or visual evidence, developers can move from being a bottleneck to being an architect of automated workflows. Success in this field is not about providing the model with more information, but about providing the right, focused information (gotchas) and measuring performance through constant, automated evaluations.