Security and stability with test driven AI coding

Key Concepts

• Vibe Coding: Utilizing AI coding agents with specific constraints and context to produce stable software.
• DORA (DevOps Research and Assessment): Research focusing on high-performing software development teams, emphasizing speed and stability.
• Pervasive Security: Integrating security considerations into all phases of the development lifecycle.
• Context Engineering: Providing AI agents with detailed information about the project, architecture, and team conventions.
• Plan-Read-Green-Refactor: A workflow for AI-assisted development emphasizing planning, test-driven development, small changes, and iterative improvement.
• Fitness Function: The test suite that defines the desired behavior of the code. Maintaining its integrity is crucial.
• Batch Size: The amount of code changed in a single commit; smaller batches are preferred for review and stability.

Leveraging DORA Principles for Stable Software with AI: A Vibe Coding Approach

This presentation explores how to leverage AI coding agents – termed “vibe coding” – to build stable and high-quality software, drawing heavily from DevOps Research and Assessment (DORA) principles. The core argument is that AI’s potential can be realized not through unconstrained code generation, but through carefully managed context and a disciplined workflow mirroring best practices for human developers. Recent research indicates that simply adopting AI doesn’t guarantee improved software delivery or stability; in fact, it can increase issues due to large, difficult-to-review changes.

The Myth of Speed vs. Stability & the Power of Pervasive Security

DORA research demonstrates that high-performing teams don’t sacrifice stability for speed; they excel at both simultaneously. This performance is rooted in “pervasive security” – integrating security objectives into every stage of development, from design to testing. This approach allows teams to spend up to 50% less time fixing security issues. The key takeaway is that security isn’t an afterthought, but a fundamental component of the development process.

Context is King: Enabling Effective Vibe Coding

The foundation of effective vibe coding is providing the AI agent with sufficient context. The analogy is drawn to a new developer versus an experienced one; a ramped-up developer, familiar with APIs, requirements, and team conventions, will produce far superior code than someone starting from scratch. This context is achieved through “context engineering,” defining architectural constraints and a directive to prioritize test-driven development. Tools like Gemini MD, used with the Gemini CLI or Anti-Gravity, facilitate this by providing instructional context to the agent.

The Plan-Read-Green-Refactor Workflow

The recommended workflow for vibe coding consists of four phases:

1. Plan: The agent articulates the design and plan upfront. Complex behaviors are broken down into separate stages.
2. Read (Red): The agent writes a single failing test based on user needs, not implementation details. The test’s failure confirms it covers new behavior. The ideal test should remain valid even with significant underlying code changes, as users care about functionality, not implementation.
3. Green: The agent’s sole objective is to make the single failing test pass. This constraint prevents two critical issues:
   • Loss of the Fitness Function: Preventing the AI from simultaneously modifying tests and code, which could lead to tests being altered to match flawed implementations.
   • Out-of-Control Batch Sizes: Limiting the amount of code generated in a single step, reducing review effort and the risk of missed quality issues. DORA research supports the benefits of small batch sizes.
4. Refactor: This phase allows for expanding functionality or addressing identified issues. Instructions for mitigation are added to the persistent context file, ensuring scalability across the project and organization.

Guardrails for High-Velocity Teams

Several guardrails are essential for mitigating risk and maintaining quality:

• Strong Version Control: Essential for rolling back changes, providing a safety net against AI-induced disruptions. The Gemini CLI offers automatic checkpointing for easy rollbacks.
• Sandboxing: Containing the agent’s actions to a limited part of the system using the Gemini CLI’s sandbox feature, particularly for potentially unsafe shell commands.
• Security Extensions: Utilizing security extensions (like those from security partners) within the Gemini CLI to identify vulnerabilities in code changes and pull requests, integrating security checks earlier in the process.
• Dedicated Code Analysis & Dependency Scanning: Acknowledging that AI-driven vulnerability detection isn’t a replacement for traditional security tools.

Data & Research Findings

• DORA Research: Demonstrates that high-performing teams achieve both speed and stability.
• Pervasive Security: Teams integrating security into all phases of development spend 50% less time remediating security issues.
• Small Batch Sizes: DORA research supports the idea that working in small batches improves product performance and reduces friction for AI-assisted teams.

Notable Quote

“Your users don't care how your app works. They care that it works.” – Emphasizing the importance of user-focused testing and the irrelevance of implementation details to end-users.

Synthesis & Conclusion

Vibe coding, when implemented with a focus on context, disciplined workflows (Plan-Read-Green-Refactor), and robust guardrails, offers a promising path to leveraging AI for stable and high-quality software development. The key is to adapt proven DevOps principles – particularly those highlighted by DORA research – to guide the interaction with AI agents. This approach prioritizes small, iterative changes, test-driven development, and pervasive security, ultimately mitigating the risks associated with large-scale, unconstrained code generation. The presenter encourages community engagement and sharing of experiences to further refine these practices. Links to context files, Anti-Gravity, Gemini CLI, and security extensions are provided for further exploration.