Your AI agent still has no memory? Fix it with these 3 patterns

Key Concepts

• Agent Memory: The capability of AI agents to retain and utilize information across interactions to improve performance.
• Callbacks: Hooks that intercept the agent's lifecycle to perform automated tasks (like updating memory) without increasing the agent's core complexity.
• Structured Data: Organizing information into defined fields (e.g., user profiles) rather than relying solely on unstructured chat logs.
• Multi-modal Memory: The ability of an agent to process and recall information from various media types, including images, audio, and video.
• Vertex AI Agent Engine: The platform/framework used for implementing these memory patterns.

---

1. Callbacks: Automated Context Management

Callbacks allow developers to implement a "spy" mechanism that observes the conversation and updates memory automatically.

• Mechanism: Custom logic is injected into the agent’s lifecycle—before or after the model is called, or before/after tools are executed.
• Example: In a trip-planning app, if a user visits a museum, the callback records this activity. When the user asks for another suggestion, the agent checks the "last activities" log and proactively excludes museums to avoid redundancy.
• Implementation: The after_tool callback updates a context variable (e.g., activity_type). The agent’s instructions are then configured to read this variable to filter future tool outputs.

2. Custom Tools: Structured Data Persistence

Moving from unstructured text to structured data allows agents to maintain persistent, reliable user profiles.

• Mechanism: Instead of relying on the agent to "remember" facts from chat history, the agent is given specific tools to save_user_preference and recall_user_preferences.
• Example: If a user mentions they are vegan, the agent uses the save tool to store this in a database as a dictionary object. In future sessions, the agent uses the recall tool to retrieve this preference, eliminating the need for the user to repeat themselves.
• Benefit: This ensures high-fidelity data retrieval and reduces the token overhead of parsing long, unstructured conversation histories.

3. Multi-modal Memory: Beyond Text

Humans perceive the world through multiple senses; multi-modal memory enables agents to store and recall non-textual information.

• Mechanism: Using the Vertex AI Agent Engine, developers can store images, videos, and audio files in a "memory bank."
• Example: A user uploads a photo, a video, and an audio clip representing their travel preferences. The agent uses a preload_memory tool to ingest these files, allowing it to synthesize a recommendation based on the "vibe" or content of those media files.
• Application: This allows for a more natural, human-like interaction where the agent understands context derived from visual and auditory inputs.

---

Logical Connections and Synthesis

The six patterns of agent memory (three previously discussed: session state, multi-agent state, and persistent memory; and three discussed here: callbacks, custom tools, and multi-modal memory) form a hierarchy of complexity:

1. Callbacks handle the process of updating memory dynamically.
2. Custom Tools handle the structure and reliability of stored data.
3. Multi-modal Memory expands the scope of what an agent can perceive and recall.

Conclusion: Building impressive AI agents requires moving beyond simple model prompting. By implementing these three advanced memory patterns, developers can create agents that act as proactive, context-aware assistants capable of managing structured user data and interpreting multi-modal inputs. These techniques shift the burden of memory management from the user to the agent, resulting in a more seamless and personalized user experience.