Autonomy Is All You Need – Michele Catasta, Replit

Key Concepts

• Autonomy in Coding Agents: Moving beyond assistance and completion towards agents capable of independent task execution.
• Semi-Sync Value: A plot illustrating the trade-off between latency/user control and agent autonomy.
• Supervised vs. Whimo Autonomy: Supervised autonomy (like Tesla FSD) requires user oversight; Whimo (Want a Human In, Maybe Out) aims for complete independence.
• Reducible Runtime: Maximizing the time an agent can operate without requiring user technical input.
• Verification: Ensuring the local correctness of code generated by the agent at each step.
• Context Management: Effectively handling and utilizing information within the agent's context window for long-running tasks.
• Sub-Agent Orchestration: Breaking down complex tasks into smaller, manageable sub-tasks handled by specialized agents.
• Parallelism: Utilizing multiple agents concurrently to accelerate task completion, balanced against the challenges of merging results.

The Pursuit of Autonomy for Non-Technical Users in Coding Agents

The presentation focuses on Raplet’s approach to building coding agents designed for users without technical expertise, framing autonomy as the central goal. The speaker argues that current agent development is shifting towards true autonomy, but a crucial dimension – autonomy for non-technical users – remains largely unexplored.

The Landscape of Agent Autonomy

The speaker begins by referencing Zix’s “semi-sync value” plot, which visualizes the relationship between latency, user control, and agent autonomy. The plot highlights three phases: low-latency interaction requiring expert knowledge, a middle ground of limited autonomy, and the current trend towards agents capable of running for extended periods (several hours). However, the speaker contends that a third dimension is needed: building agents that are truly autonomous for non-technical users.

Two Types of Autonomy: Supervised vs. Whimo

A key distinction is made between two types of autonomy. Supervised autonomy is exemplified by Tesla’s Full Self-Driving (FSD) – requiring a licensed driver ready to intervene. This mirrors current coding agents that demand technical proficiency. Raplet is focused on Whimo autonomy (“Want a Human In, Maybe Out”), aiming for an experience where users require no technical knowledge, akin to not needing a “driving license” to use the agent. The goal is to empower all knowledge workers to create software without needing to understand the underlying technical complexities.

From Assistance to Autonomy: A Generational Shift

The evolution of coding agents is described in three generations:

1. Completion/Assistance: Early AI-powered tools focused on short feedback loops and assisting with code snippets.
2. React-Based Agents: Initial attempts at autonomy using the React framework layered on top of Large Language Models (LLMs).
3. Autonomous Agents (Rapid Agent v3): Breaking the one-hour autonomy barrier, capable of handling long-horizon tasks with sustained coherence. Rapid Agent v3 showcases these properties.

Redefining Autonomy & The Importance of Scope

The speaker challenges the conventional definition of autonomy as simply “long run time” or “loss of control.” Instead, autonomy should be scoped – focusing on automating technical decisions while preserving user control over the overall goals. Rapid Agent 3 achieves this by handling all technical decisions, potentially leading to longer runtimes, but only when tackling broad tasks. Narrowly scoped tasks allow for fast, autonomous execution without sacrificing user control. The user cares about the outcome, not the technical implementation. Autonomy shouldn’t be a “vanity metric” but a means to maximize the “reducible runtime” – the time the agent operates without requiring user technical input.

Pillars of Autonomy

Three pillars are crucial for achieving true autonomy:

1. Frontier Model Capabilities: Leveraging powerful LLMs like those developed by the audience (implicitly acknowledging the contributions of other AI developers).
2. Verification: Rigorous testing at each step to ensure local correctness, preventing the accumulation of errors (“shaky foundations”). Without verification, agents build “painted doors” – features that appear functional but are broken.
3. Context Management: Balancing global coherence (alignment with user intent) with the ability to manage both high-level goals and individual tasks.

Addressing the "Painted Door" Problem: Autonomous Testing

A significant challenge is the prevalence of broken features (“painted doors”) in code generated by agents. Internal evaluations at Raplet revealed that over 30% of individual features are initially broken. Users, especially non-technical ones, are frustrated by these issues. The solution lies in autonomous testing.

• Why Autonomous Testing? It breaks the feedback bottleneck (users are unwilling to perform tedious manual testing), prevents error accumulation, and overcomes the “laziness” of LLMs (verifying that a task is truly completed).
• Approaches to Code Verification: The speaker outlines a spectrum from static code analysis (LSPs) to generating and running unit tests, API testing, and ultimately, autonomous browser-based testing.
• Playwright Integration: Raplet utilizes Playwright, a framework for reliable end-to-end testing, to interact with applications programmatically. Writing Playwright code directly is more manageable for LLMs and allows for reusable regression tests. This approach is significantly cheaper and faster than computer vision-based testing.

Context Management & Sub-Agent Orchestration

While long context windows are becoming available, the speaker argues they aren’t necessary for long-running tasks. Effective context management can achieve similar results within more manageable context limits (around 200,000 tokens). Key techniques include:

• Using the Codebase as State: Writing documentation and task lists directly into the code.
• Persisting State on the File System: Offloading memory requirements.
• Dumping Memories to Disk: Retrieving relevant information only when needed.

Sub-agent orchestration is highlighted as a critical component. Breaking down tasks into smaller, independent sub-tasks handled by specialized agents improves context management, reduces memory requirements, and enhances overall performance. This mirrors the software engineering principle of “separation of concerns.”

The Future: Parallelism and the Core Loop as Orchestrator

While long runtimes are valuable, the speaker argues that parallelism is crucial for enhancing the user experience. Running multiple agents concurrently accelerates task completion, but introduces the challenge of merging results and resolving conflicts.

Raplet’s future direction involves shifting the orchestration of parallel tasks from the user to the core loop of the agent. This means the agent will automatically decompose tasks and manage parallelism, relieving the user of cognitive burden and potentially mitigating merge conflicts through intelligent task allocation. This approach aims to make the agent experience more dynamic and engaging.

Conclusion

The presentation concludes with a call for continued innovation in autonomous agent development, emphasizing the importance of verification, context management, and parallelism. The ultimate goal is to empower all knowledge workers to create software without needing to be technical experts, abstracting away complexity and fostering creativity.