Beyond the Prototype: Using AI to Write High-Quality Code - Josh Albrecht, Imbue

Key Concepts

• AI Agents for Coding
• Software Quality (Defect-based)
• Sculptor (Imbue's experimental coding agent environment)
• Preventing Problems in AI-Generated Code
• Detecting Problems in AI-Generated Code
• Strict Style Guides
• Linting
• Testing (Unit, Integration, Happy/Unhappy Paths)
• Test Coverage
• Sandboxing
• LLM-based Code Analysis and Fixing

Introduction to Sculptor and the Problem

Josh Albertch, CTO of Imbue, introduces Sculptor, an experimental coding agent environment designed to improve the reliability and trustworthiness of AI-generated code. The core problem Sculptor addresses is the gap between the code produced by AI coding tools and the requirements for production-ready code, especially in established codebases. The goal is to provide a third option beyond manually reviewing every line of code or blindly merging, by leveraging AI to build trust in the generated code.

Defining Software Quality and the Development Process

Imbue defines software quality based on the number of defects. The aim is to identify and address problems as early as possible in the development process. Sculptor is designed to provide synchronous and immediate feedback on code changes, unlike pull request review tools that operate later in the process.

Preventing Problems in AI-Generated Code

Four key strategies for preventing problems are discussed:

1. Learning: Sculptor facilitates research and information gathering about existing technologies and solutions to avoid redundant work.
2. Planning: The system encourages users to start by planning. Users can modify the system prompt to force the AI agent to create a plan before writing any code. This can be achieved by changing the system prompt to force the AI agent to first make a plan without writing any code at all.
3. Writing Specs and Docs: Sculptor helps detect when code and documentation become outdated, reducing the barrier to maintaining documentation. It can also highlight inconsistencies within specifications.
4. Strict Style Guide: Enforcing a strict style guide, including suggestions to promote functional, immutable code, helps prevent certain classes of errors. Imbue is developing a custom style guide tailored for AI agents.

Detecting Problems in AI-Generated Code

Three methods for detecting problems are outlined:

1. Running Linters: Sculptor integrates with linters (e.g., Ruff, MyPy, Pylint) to automatically detect and fix errors. The system understands which issues existed before and after the AI's changes, preventing the introduction of new errors.
2. Writing and Running Tests: The ease of generating tests with AI removes the traditional objections to writing tests.
   • Functional Style: Writing code in a functional style (no side effects) makes it easier for LLMs to analyze and validate code.
   • Happy/Unhappy Unit Tests: Focus on "unhappy" unit tests to find bugs. LLMs can generate numerous inputs and identify unexpected outputs.
   • Test Suite Management: Consider discarding or refactoring generated unit tests to avoid confusing the LLM when behavior changes.
   • Integration Tests: Emphasize integration tests that verify the program's behavior from the user's perspective. AI systems can excel at writing these tests, especially with well-defined test plans.
   • Test Coverage: Prioritize test coverage to ensure that changes are thoroughly tested.
   • Sandboxing: Run tests in sandboxes without secrets to prevent unintended consequences and flaky tests.
3. Asking an LLM: LLMs can check for various issues, including inconsistencies with specifications, violations of style guides, and missing details. Sculptor aims to allow users to extend these checks with custom best practices.

Fixing Issues

The presentation emphasizes that fixing issues is often easier than expected, especially with well-defined problems. "A problem well-stated is half-solved." Simple strategies like multiple attempts with different agents can be effective. Good sandboxing enables parallel execution of numerous agents to find a successful solution.

Future Tools and Conclusion

The presentation concludes by highlighting the upcoming tools for debugging, logging, tracing, profiling, automated quality assurance, code generation from visual designs, and improved contextual search. These advancements, combined with better AI models, promise to significantly improve the development experience. Imbue encourages collaboration and integration of these tools into Sculptor. The main takeaway is that the development experience will become much easier as all these tools work together.