Breaking the Chain: Agent Continuations for Resumable AI Workflows - Greg Benson

Agent Continuations: Managing Agent State and Human-in-the-Loop Processing

Key Concepts:

• Agent Continuations: A mechanism for capturing the full state of complex agents, enabling human-in-the-loop processing and reliable agent continuation through snapshots.
• Human-in-the-Loop: Incorporating human approval or decision-making into agent workflows, especially for high-value or high-risk tasks.
• State Management: Saving the current state of an agent's execution to resume later, particularly important for long-running agents and handling failures.
• Agent Loop Persistence: The requirement for the agent's core execution loop to remain active, even when waiting for external input (e.g., human approval).
• Messages Array: A log of interactions between the agent and the LLM, used to maintain context and history for subsequent inferences.
• Continuation Object: A data structure containing the agent's state, including the messages array and metadata for resuming execution.
• Sub-Agents: Nested agents within a main agent, allowing for more complex and modular agent architectures.

Challenges in Agent Execution

• Human Approval: Ensuring human oversight for critical agent actions, such as transferring money or deleting accounts.
• Rate Limiting: Handling limitations on LLM API usage due to model popularity or other factors.
• Long-Running Agents: Mitigating the risk of failure in agents performing complex tasks over extended periods.
• Distributed Environments: Addressing considerations for running agents in scalable, distributed systems.

Basic Agent Execution

• Agents operate in a loop: LLM call -> Tool call (if needed) -> LLM call -> Tool call, and so on.
• The LLM determines which tools to use based on the user's request and the agent's configuration.
• Tools can themselves be agents (sub-agents), creating nested agent architectures.

Agent Execution Scenarios

• Tool Calls Requiring Human Approval: Agents need to pause and request human authorization before executing certain tool calls.
• Rate Limiting: Agents need to handle rate limits gracefully, potentially by pausing and resuming later.
• Long-Running Agents: Agents need to be able to checkpoint their state and resume from that point in case of failure.

Agent Loop Persistence Problem

• Traditional agent frameworks require the agent loop to remain active even when waiting for human input.
• Agent continuations aim to eliminate this requirement by allowing the agent loop to be shut down and restarted later.

Agent Continuations Explained

• Inspired by the programming language concept of continuations, which allows capturing and resuming program execution at a specific point.
• Agent continuations enable pausing agent execution, saving the state, and resuming later, even across multiple tool calls, LLM calls, and sub-agents.
• The goal is to support human approval workflows and persist agent state in case of failures.

Implementation Details

• Leverages the existing "messages array" used by LLMs to maintain a history of interactions.
• Creates a "continuation object" that embeds the messages array along with additional metadata for resuming execution.
• The continuation object contains information about which tool call to resume at and whether human approval has been granted.
• The framework extracts core information from the continuation object to make it easy for the application layer to process human approvals.
• Once the application layer updates the continuation object, it is sent back to the agent, which reconstructs its state and continues execution.
• A key advantage is that agent loops can be shut down after creating the continuation object, eliminating the need for continuous persistence.

Continuation Object Structure

• Single-Level Continuation: Contains the messages array, resume request (specifying the tool call to resume at), and a "processed" field indicating approval status.
• Multi-Level Continuation: Uses a recursive format to handle nested agents, with each level containing its own messages array, resume request, and processed field.

Example: Multi-Level HR Agent

• A top-level HR agent uses an email tool and a sub-agent called the "account agent."
• The account agent has tools for creating and authorizing accounts.
• The "authorize account" tool requires human approval.
• When the agent reaches the "authorize account" tool, it suspends execution and creates a continuation object.
• The continuation object propagates back to the application layer, which presents the approval request to the user.
• Once the user approves, the updated continuation object is sent back to the HR agent, which restores its state and continues execution.

Demo

• The demo showcases the decoupling of agent execution and state management.
• A prompt is sent to the agent, which suspends execution and generates a continuation object.
• The continuation object is saved to a file, demonstrating that it contains all the necessary state information.
• The continuation object is manually edited to simulate human approval.
• The edited continuation object is sent back to the agent, which reconstructs its state and completes the task.

Prototype and Future Directions

• The prototype is built on top of the OpenAI Python API with no other dependencies.
• Future work includes implementing more general agent suspension mechanisms beyond human approval.
• The goal is to extend existing agent frameworks like Strands or Pylantic AI, rather than creating a new framework from scratch.
• Agent Creator, Snaplogic's visual agent building interface, also supports agent continuations.

Conclusion

Agent continuations offer a novel approach to managing agent state and incorporating human-in-the-loop processing. The prototype implementation is available on GitHub. This mechanism addresses limitations in existing frameworks by providing a robust solution for human approval and handling complex, nested agent architectures.