Stateful environments for vertical agents — Josh Purtell, Synth Labs

Key Concepts

• Stateful Environments: An engine that computes results external to the agent implementation, providing a manageable representation of a complex system for the agent to interact with.
• Agent: An AI entity that interacts with and manipulates the stateful environment.
• Environment: A containerized logic behind a task, separate from the AI algorithm, allowing the AI to perform actions without explicit instructions.
• Network Boundaries: Separating the agent's process from the stateful environment's process, enabling asynchronous work, multi-agent systems, and improved reliability.
• Rollbacks: The ability to revert the environment to a previous state, useful for correcting errors or exploring alternative paths.
• Language Agent Tree Search: A technique where the agent explores multiple paths in the environment, evaluates their outcomes, and converges on the most promising one.

Stateful Environments for AI Agents: A Deep Dive

Introduction

Josh, the founder of Synth, discusses the importance of structuring agent code using stateful environments, particularly for vertical applications like finance, accounting, and health. He emphasizes that this approach improves agent performance and simplifies development.

Defining the Environment

The concept of an "environment" has a history in reinforcement learning (RL). RL aims to train AI to perform tasks without explicitly programming how to do them. Environments serve to isolate the task's logic from the AI algorithm.

• Early Implementations: RL-Glue was an early implementation.
• OpenAI Gym: OpenAI (initially an RL company) introduced OpenAI Gym.
• Agent Computer Interface: Academic papers like SUB bench and SU agent coined the term "agent computer interface" for vertical applications.

The idea is to build upon existing concepts rather than reinventing the wheel.

The Need for Stateful Environments

Two years ago, simple tasks for language models (LMs) didn't require complex abstractions. However, as models improved (especially with models like Sonnet 35) and were used with more effective tools (like API-based tools), the need for robust abstractions became apparent.

• Claude Artifacts: The introduction of Claude artifacts highlighted the need for agents to work on and iterate on products within a web application.

What is a Stateful Environment?

A stateful environment is an engine that computes results external to the agent's implementation. The agent interacts with the environment, but the underlying logic (e.g., API access, Excel manipulation) is handled separately.

• Representation: The environment exposes a simplified representation to the agent, hiding unnecessary complexity (e.g., showing only relevant terminal information instead of the entire OS).
• Network Boundaries: Stateful environments often have network boundaries, allowing the agent to run in a separate process. This is crucial for RL training and multi-agent systems.

Benefits of Stateful Environments

Stateful environments offer several advantages for improving AI agents:

• Simplified Agent Revamping: Containerizing the application logic makes it easier to update the agent when new models are released.

• Multi-Agent Systems: Network boundaries enable multiple agents to work on the same product asynchronously and reliably.

• Rollbacks: The ability to reset the environment to a previous state allows for easy error correction and exploration of alternative paths.

• Language Agent Tree Search: Resetable environments facilitate language agent tree search, where the agent explores multiple paths, evaluates their outcomes, and converges on the best one.

  • Example: In Minecraft, the agent can branch out in different directions, evaluate the results, and choose the most successful path. This is particularly useful in long-horizon tasks where avoiding derailment is crucial.

Language Agent Research

A few years ago, there was a paper called language agent research, that was, you know, really impressive and it got really good results, but it's almost impossible to implement in production because just nobody had really good um, abstractions for it.

Practical Implementation

Synth provides an open-source repository with implementations of stateful environments across various academic benchmarks.

• GitHub: Find the repository by searching for "synth AI environments."

Conclusion

Stateful environments are a valuable abstraction for building effective AI agents, especially in complex, vertical applications. They offer benefits such as simplified agent management, support for multi-agent systems, and the ability to perform rollbacks and tree search. By containerizing the application logic and providing a manageable representation to the agent, stateful environments enable more robust and reliable AI solutions.