Automating Large Scale Refactors with Parallel Agents - Robert Brennan, AllHands

Key Concepts

• AI-Assisted Software Engineering Evolution: Progressing from context-aware code snippets (Copilot) to autonomous agents (Devon, Open Hands) and now parallel agent orchestration for large-scale refactoring.
• Agent Orchestration: Utilizing multiple agents, potentially creating micro-agents, to tackle complex tasks like tech debt remediation and code modernization.
• Open Hands Philosophy: Commitment to open-source development and community-driven AI in software engineering, recognizing the transformative impact of LLMs.
• Agent Deployment Models: Plugins, AI-Powered IDEs, Local Agents, Cloud-Based Agents, and Orchestration, with a shift towards cloud-based and orchestrated solutions for complex tasks.
• Open Hands SDK: A tool for building agents capable of interacting with remote workspaces (Docker containers) to perform tasks like vulnerability scanning and code modification.
• Agent Trajectory: The sequence of actions (tool calls) and observations (outputs) that define an agent’s execution path.

The Rise of Agent Orchestration

The field of AI-assisted software engineering is rapidly evolving. Initially, Large Language Models (LLMs) offered context-aware code snippets, like those found in GitHub Copilot, boosting developer productivity. This progressed to autonomous coding agents, such as Devon and Open Hands’ own agents, capable of writing, executing, and debugging code. The current frontier is agent orchestration, where multiple agents work in parallel to address complex challenges, particularly substantial tech debt. Open Hands, founded in response to the Devon demo, champions an open-source, community-driven approach to this evolution, believing the impact of LLMs will fundamentally reshape the role of a software engineer.

Deployment & Use Cases

Agent deployment exists across a spectrum: plugins integrated into IDEs, AI-powered IDEs, local agents for ad-hoc tasks, and cloud-based agents offering scalability and security. Open Hands focuses on cloud-based agents and orchestration due to their isolation and suitability for complex tasks. While most developers currently utilize single, local agents, a growing number are experimenting with orchestration. Practical use cases include basic code maintenance (like resolving CVEs – with a client example showing a 30x improvement in resolution time), automating documentation, code modernization (adding type annotations, splitting monoliths), and migration tasks (Spark 2 to Spark 3, React with Redux to Zustand).

Challenges in Large-Scale Automation

Despite the potential, significant challenges remain. These include the limited context window of agents, the tendency towards “laziness” resulting in incomplete solutions, a lack of domain knowledge, error compounding across iterations, the difficulty of translating human intent into agent instructions, the complexity of decomposing tasks, and the challenge of defining clear “done” criteria. A workflow for orchestrated tasks involves task decomposition, parallel agent execution, collation & merging of outputs, and crucial human review – aiming for approximately 90% automation with ongoing oversight.

Practical Demonstration: Code Smell Elimination

A demonstration showcased eliminating code smells from the Open Hands codebase using the Open Hands SDK. This involved visualizing code dependencies, batching files based on those dependencies, utilizing a verifier agent to identify smells, and a fixer agent to generate pull requests for corrections, followed by human review and merging.

Building Agents with the Open Hands SDK

The Open Hands SDK allows developers to create agents that interact with remote workspaces, specifically Docker containers. A demonstration illustrated building an agent to scan a GitHub repository for vulnerabilities related to open-source Component Analysis (CLA). This process involved five prompts, emphasizing SDK functionality over optimal coding practice. The agent clones a repository into a Docker container (started with docker run <agent server>), scans for vulnerabilities, saves the results to a vulnerabilities.json file, and retrieves the file for analysis. The agent operates through a series of events – actions (tool calls to the LLM) and observations (outputs). The demonstration highlighted the use of a Large Language Model (LLM), a terminal tool, and a file editor tool within the agent’s workflow. The presenter acknowledged that a more structured approach, utilizing separate files and pre-built codebases, would be preferable in a production environment. Installation issues with the CLI, specifically the -AI flag, were noted, with suggestions to use an executable binary or uv run as alternatives.

Technical Foundations

Key technical terms include CVE (Common Vulnerabilities and Exposures), SDK (Software Development Kit), LLM (Large Language Model), Docker (for containerization), Workspace (the Docker container), and Trajectory (the agent’s action/observation sequence). The demonstration successfully identified vulnerabilities in a test repository, showcasing the SDK’s capabilities.

Conclusion

The presented material demonstrates a significant shift in software engineering towards AI-assisted automation, particularly through agent orchestration. While challenges remain regarding context, error handling, and human oversight, the Open Hands SDK provides a powerful tool for building agents capable of tackling complex tasks like vulnerability scanning and code refactoring. The emphasis on open-source development and community collaboration suggests a future where AI increasingly augments, rather than replaces, the role of the software engineer.