How to build reliable software with AI agents

Key Concepts

• Cognitive Debt: The erosion of personal problem-solving skills and memory caused by over-reliance on AI agents.
• Cognitive Surrender: The state where a developer stops thinking critically and blindly accepts AI-generated output.
• Comprehension Debt: The lack of understanding regarding how a codebase is constructed, leading to an inability to debug or maintain systems when AI fails.
• Human-in-the-Loop (HITL): The necessity of maintaining human oversight and judgment in the software development lifecycle.
• Generative Engineering: The practice of using AI agents to automate the creation of code and features.

---

The Tension in Modern Software Development

The developer community is currently experiencing a significant debate regarding the role of AI. While productivity gains are evident, there is growing concern that AI is increasing technical debt, eroding the "craft" of programming, and leading to a loss of fundamental engineering skills. The consensus among many developers is that AI agents should focus on reducing long-term maintenance costs rather than simply increasing the speed of code production.

The Spectrum of AI Implementation

The approach to using AI agents varies significantly based on the context of the project:

• Small-Scale/Greenfield Projects: Solo founders or small teams building new, "evergreen" applications may adopt a more experimental or "Wild West" approach to AI, as the risk profile is lower.
• Large-Scale/Legacy Systems: Teams maintaining complex, user-facing applications must prioritize stability and reliability. In these environments, unchecked AI output is dangerous, and rigorous verification processes are mandatory.

Risks of Over-Reliance

The author highlights a progression of negative outcomes when developers rely too heavily on AI:

1. Erosion of Skills: Current tools are optimized for "goal completion" (getting to the end product) rather than "learning" (understanding the architecture).
2. Cognitive Debt: As developers defer tasks to agents, they lose the mental models required to understand how their software is constructed.
3. Cognitive Surrender: The final stage where the developer stops questioning the AI’s output entirely. This creates a critical vulnerability: if the agent fails or production goes down, the developer lacks the foundational knowledge to diagnose or fix the issue.

Framework for Responsible AI Integration

To maintain high-quality software while leveraging AI, the author suggests the following:

• Verification Protocols: Developers must establish clear definitions of "good" code. This includes implementing extensive unit testing, visual regression testing, and maintaining a clear architectural vision to ensure the AI does not "go off the rails."
• Intentional Code Review: Code review remains essential. It should not be bypassed simply because an agent wrote the code.
• The 70/30 Rule: AI agents can realistically handle 70–80% of the workload. The remaining 20–30% requires human judgment, architectural oversight, and a deep understanding of software fundamentals.

Notable Quotes

• "If you're using an agent to build your software... you as a human, as a senior engineer, need to be able to verify that output is correct."
• "Cognitive surrender is where you're no longer critical thinking, you're just deferring entirely to your agents."
• "I have personally never felt more productive and also never more tired."

Conclusion

While generative engineering offers immense productivity potential, it introduces a "hidden cost" in the form of cognitive and comprehension debt. The primary takeaway is that AI should be treated as a tool for augmentation, not a replacement for human expertise. Developers must remain "in the loop" to ensure that the final 20–30% of any project—the critical judgment and quality assurance—is handled by a human who understands the underlying system.