Stateful Agents — Full Workshop with Charles Packer of Letta and MemGPT

Key Concepts

• Stateful Agents: Agents that maintain memory and learn from experience, contrasting with stateless LMs.
• LMOS (LLM Operating System): A memory management system for LLMs, enabling them to retain and utilize information over time.
• Context Management: The process of optimally arranging the context window of an LLM, drawing from a larger state.
• Core Memory: Top-level, readily accessible memory within the context window.
• Archival Memory: External data sources (e.g., vector databases) that the agent can access to "jog its memory."
• Recall Memory: Conversation history that the agent can specifically search.
• Tool Calling: The mechanism by which agents interact with external tools and manage memory.
• Heartbeat Requests: A mechanism where agents explicitly signal their intent to continue processing, enabling indefinite chaining.
• Multi-Agent Systems: Systems where multiple stateful agents interact and communicate, often asynchronously.

1. Introduction and the Problem of Statelessness

The workshop focuses on the concept of "stateful agents" as a solution to the limitations of current LLM-driven agents. The speaker argues that while agents are often defined as LMs taking actions in a loop, a crucial aspect is often overlooked: the agent's ability to update its state within that loop. Because LMs are inherently stateless (transformers), a mechanism for updating state is essential.

• Problem: Current agents struggle to learn from experience, hindering their ability to function as assistants, companions, or co-pilots.
• Example: The Valentine's Day example illustrates the devastating error an agent can make if it lacks a permanent memory store and relies solely on context.
• Key Argument: Stapleness (memory) is the most important thing to solve if agents are to deliver on their potential.

2. MEGPD: A Memory Management System for LLMs

The speaker introduces MEGPD, a memory management system designed to address the statelessness of LLMs. The core idea is that if LMs are improving, memory management should be handled by another LM, allowing the AI to manage its own memory.

• Stateless vs. Stateful: The speaker contrasts the common approach of appending to a list in process memory with a system where context is compiled from a larger state.
• Context Compilation: The process of optimally arranging the context window, drawing from a larger state. This is what power users of chatGPT or Claude do manually.
• Goal: Automate context compilation with a machine (an LLM).

3. Why Stateful Agents Matter

The speaker outlines the key reasons why stateful agents are crucial for advancing AI:

• Learning from Experience: Current agents cannot effectively learn from experience, limiting their ability to adapt and improve over time.
• Data Utilization: Enterprises have vast amounts of data that cannot fit into the context window. Stateful agents can learn from this data over time, effectively undergoing a "post-training" phase.
• Human-like Intelligence: The speaker emphasizes the importance of creating AI that is more human-like, particularly in how it deals with memory.

4. The Promise of Stateful Agents

Stateful agents offer several key advantages:

• Improved Product Experience: Eliminating derailment and ensuring that the AI gets better over time as it learns more about the user.
• More Human-like Behavior: Creating agents with fuzzy memory, forgetfulness, and recall, mimicking human memory constructs.

5. Workshop Structure and Tools

The workshop is divided into two parts:

• Part 1: Jupyter Notebook: Laying out the basic ideas behind a context management system. The core concept is making an LM aware of the context problem and providing it with tools to manage memory.
• Part 2: Leta Framework and AD: Building stateful agents using the Leta framework and exploring a new iteration of the playground.

6. Notebook Walkthrough: Building a Basic Stateful Agent

The speaker walks through a Jupyter notebook demonstrating the core concepts of stateful agents using the Leta framework.

• Server-Client Architecture: Leta operates on a server-client architecture, where agents are intended to be staple and persist indefinitely.
• Memory Blocks: The fundamental unit of memory in a Leta agent, consisting of strings with associated handles.
• Agent Creation and Messaging: Demonstrating how to create an agent, define its persona and memory blocks, and send messages to it.
• Message Types: Introducing different message types (reasoning message, assistant message, tool call message, etc.) that are used to interact with the agent.
• Unpacking Agent State: Showing how to access and inspect the agent's system prompt, tools, and memory blocks.
• Core Memory Manipulation: Demonstrating how the agent can use tools to edit its own core memory, such as updating the user's name.
• Archival Memory: Introducing the concept of archival memory as an external data store that the agent can access.
• Custom Tools: Showing how to create and attach custom tools to the agent, allowing it to perform specific actions.

7. Leta AD: A Visual Interface for Building Stateful Agents

The speaker introduces the Leta AD, a visual interface for building and managing stateful agents.

• Low-Code Environment: The AD provides a faster and more intuitive way to iterate on agent design compared to using an SDK.
• Agent Creation and Configuration: Demonstrating how to create an agent, set its system prompt, define memory blocks, and configure its context window.
• Tool Management: Showing how to attach and manage tools, including custom tools written in Python.
• Context Simulator: Introducing a feature that allows developers to visualize the full payload being sent to the LM, providing insights into context management.

8. Multi-Agent Systems: Communication and Coordination

The speaker explores the concept of multi-agent systems, where multiple stateful agents interact and communicate.

• Asynchronous Communication: Emphasizing the importance of asynchronous communication between agents, mimicking human interaction in remote companies.
• Multi-Agent Tools: Introducing tools that enable agents to send messages to each other, either synchronously or asynchronously.
• Supervisor-Worker Pattern: Describing a common multi-agent pattern where a supervisor agent delegates tasks to worker agents.

9. Cookbook Example: Multi-Agent Message Passing

The speaker walks through a cookbook example demonstrating multi-agent message passing using two agents with distinct personalities and goals.

• Agent Setup: Creating two agents, Bob (grumpy and guarding a secret key) and Alice (friendly and trying to help Bob).
• Message Exchange: Demonstrating how the agents communicate with each other, with Alice trying to reach out to Bob and Bob responding defensively.
• Tool Management: Showing how to detach a tool from an agent, effectively preventing it from performing certain actions.

10. Conclusion

The workshop concludes by emphasizing the potential of stateful agents and the Leta framework for building more intelligent and human-like AI systems. The speaker encourages attendees to explore the Leta AD and experiment with multi-agent systems.

11. Q&A Highlights

• Use Cases: Verticalized agents, enterprise workflows, and any application requiring memory and state.
• Memory Management: The system is designed to remove as much human design of memory management as possible and banking on the LLM getting better and better.
• Forgetting: No built-in forgetting mechanism, but timestamps allow for consolidation.
• Data Compression: Requires manual effort to chain function calls and regenerate memory blocks.
• Active Document Editing: Recommends using Anthropic's Sweetbench tools as a reference.
• Coding Agents vs. Tool Calls: Secure execution is key for coding agents; latency is the main issue.