Stanford Global Alumni Webinar | August 2025 | AI Agent Simulation of Human Behavior

Key Concepts

• AI Agents: Simulated entities powered by AI that can replicate human behavior and interact with an environment.
• Generative Agents: A specific type of AI agent capable of generating human-like behavior and responses.
• Agent-Based Models (ABMs): Computational models that simulate the actions and interactions of autonomous agents (individuals, organizations, etc.) to understand complex systems.
• Large Language Models (LLMs): AI models trained on vast amounts of text data, capable of understanding and generating human-like text (e.g., ChatGPT, Claude, Llama).
• Retrieval Augmented Generation (RAG): A technique that enhances LLMs by retrieving relevant information from a knowledge base before generating a response.
• Memory Stream: A chronological record of an agent's observations and experiences.
• Reflection: The process by which agents form higher-level understandings of themselves, their goals, and their dispositions based on their memory stream.
• Planning: The ability of agents to strategize and outline future actions, from daily schedules to minute-by-minute tasks.
• Digital Twin: A virtual replica of a real-world entity, in this context, a generative agent representing a specific individual.
• Look Before You Launch: A framework for using simulations to anticipate potential issues and refine systems before deployment.
• Quantitative Accuracy: The degree to which simulations can accurately predict numerical outcomes or distributions.
• Qualitative Accuracy: The degree to which simulations can accurately predict attitudes, opinions, or narrative outcomes.

The "What If" Machine: Simulating Human Behavior with AI Agents

The core idea presented is the development of AI agents capable of simulating human behavior to create a "what if" machine. This machine aims to address the inherent uncertainty and incomplete information faced when making decisions in various domains, from business and policy to personal life. By simulating how people might react to different scenarios, individuals and organizations can make more informed and effective decisions.

The Problem of Incomplete Information and Bad Bets

• Decision-making challenges: Humans often make decisions based on incomplete information about how others will react. This applies to organizations (customer reactions), leaders (organizational performance), and even individuals (personal choices).
• Consequences: This incomplete information leads to "bad bets" – decisions that are ultimately wrong despite best efforts. This is not due to a lack of intelligence but the inherent difficulty of predicting future outcomes.
• Historical precedent: The challenge of predicting group behavior is not new, dating back to at least 1906 with sociologist Robert Merton's observations on collective behavior leading to unintended consequences (e.g., everyone trying to avoid crowds by going to the same vacation spot).
• Applications: This problem spans various fields, including consumer packaged goods (CPG), personal finance, product design, policy management, and academia.

The Promise of AI Simulation

• The "What If" Machine Concept: The speaker proposes the idea of a "what if" machine that could simulate potential outcomes of decisions, allowing users to explore customer reactions, failure pathways, or policy impacts before deployment.
• AI Agents as Simulators: The key to this "what if" machine lies in creating simulated AI agents that can replicate human behavior. If these agents can accurately mimic how people (customers, employees, etc.) would act, then simulations become powerful tools.
• Evolution of Simulation:
  • Agent-Based Models (ABMs): Dating back to 1978 (Thomas Schelling), ABMs have been used in economics and policy to model human behavior, even for complex issues like pandemic spread.
  • Entertainment: Games like "The Sims" demonstrate sophisticated simulations of human behavior for entertainment purposes.
  • AI Teammates: The concept of AI agents interacting with humans in a team setting requires them to reason about human responses.
  • "Look Before You Launch": This approach aims to create tools that help foresee potential problems and set rules before launching systems, preventing negative outcomes.

Limitations of Traditional Simulation Models

• Rigidity: Previous models were often too rigid:
  • Parameter-based: Reducing humans to a small number of parameters (e.g., five) is insufficient to capture the richness of human behavior.
  • Script-based: Relying on pre-written scripts (like in "The Sims") limits simulations to only what developers can anticipate, leading to incompleteness.
• Academic Conclusion: The academic literature concluded that these models were "highly stylized and have had minimal impact" in practice.

The Breakthrough: Large Language Models (LLMs)

• LLM Capabilities: Modern LLMs (ChatGPT, Claude, Llama, etc.) have been trained on vast datasets of human behavior research and social media, exposing them to the full spectrum of human actions.
• Prompting for Personas: LLMs can be prompted to adopt the perspectives of individuals with diverse backgrounds, experiences, and traits. By creating multiple such prompts, a "crowd" of simulated people can be generated.
• Generative Agents: This led to the development of "generative agents" – autonomous AI entities that play the part of different people within a simulated environment.
• "Smallville" Example: A viral demonstration involved creating a town called "Smallville" populated by 25 generative agents, each with its own persona and daily activities (e.g., an artist painting, college students attending classes). This allows for visualization of simulated human behavior.
• Market Research Potential: Research from Andreessen Horowitz highlights this as a potential next generation of market research tools, citing the "Smallville" work.

How to Build Generative Agents: A How-To Guide

The process of creating generative agents involves several key components:

1. Persona Creation:

   • Visuals: Online artists can create pixel art for character visualization (optional, for demonstration).
   • Description: Define a persona with a name, role, personality traits, and knowledge about other agents in the simulation.
   • Relationships: Specify relationships (e.g., married, family members) and their roles (e.g., college student studying music theory).
   • Initial Knowledge: Agents need to be informed about other agents they know at the start of the simulation.

2. Agent Architecture (The "Memory, Reflection, Planning" Framework):

   • Memory Stream:

     • Function: A blow-by-blow record of everything the agent observes and experiences.
     • Challenge: LLMs can get distracted by very long contexts.
     • Solution: Retrieval Augmented Generation (RAG):
       • Recency: Prioritize memories that are recent.
       • Importance: Prioritize memories that are significant to the agent.
       • Relevance: Prioritize memories pertinent to the current situation.
       • Mechanism: Acts like a "Google search" over memories to retrieve relevant information for the LLM's context window.

   • Reflection:

     • Purpose: To move beyond simple episodic memory (a log of events) and develop higher-level reflections on dispositions, interests, and goals.
     • Method: At regular intervals, agents "reflect" on memories from their stream to produce higher-level conclusions.
     • Process: Grouping memories and reinserting reflections back into the memory stream creates increasingly sophisticated self-understanding and goal-oriented behavior.

   • Planning:

     • Purpose: To enable agents to behave believably over long periods.
     • Methodology:
       • Daily Planning: Agents plan their entire day.
       • Hourly/Minute-by-Minute Breakdown: Plans are drilled down into finer temporal resolutions.
       • Reactive Replanning: When agents observe something in the environment, they are prompted to decide if a reaction is needed and to replan accordingly.

3. Grounding Actions in the Environment:

   • Natural Language Actions: Agents primarily act through natural language (e.g., "Isabella is drinking coffee").
   • Concrete Movements: These language actions need to be translated into physical movements and visualizations within the simulation environment (e.g., walking to a chair, rendering an emoji).

4. Intervention and Interaction:

   • Talking to Agents: Users can interact with agents by asking questions (e.g., "Who's running for mayor?").
   • Controlling Agents: Users can influence agent behavior by acting as their "inner voice" (e.g., "John, you're running for mayor").
   • Environmental Interventions: Users can alter the environment (e.g., setting a toaster on fire) to observe agent responses.

Case Study: The Valentine's Day Party Simulation

• Scenario: Isabella, a cafe owner, is given the intent to plan a Valentine's Day party.
• Emergent Behavior: Without explicit party-planning modules, the agent spontaneously:
  • Remembers to tell other agents about the party.
  • Recruits another agent (Maria) for decoration.
  • Leads to information diffusion patterns resembling word-of-mouth.
• Outcomes:
  • Half the town heard about the party.
  • Five agents attended, three were busy, and four expressed interest but didn't show up.
• Accuracy vs. Believability: While the outcome is "broadly plausible" and "compelling evidence of believable behaviors," it's not definitively proven to be perfectly accurate.
• Intervention Study: Researchers later introduced a simulated radio announcement about a new communicable disease.
  • Result: No one attended the party except for one agent who hadn't heard the news.
  • Control: A non-infectious disease announcement did not affect party attendance.
  • Insight: Demonstrates the power of simulations to explore "what if" scenarios with interventions.

Measuring Accuracy: Beyond Believability

• The Challenge: Believability (like in cartoons) doesn't guarantee accuracy. For decision-making, accuracy is crucial.
• Methods for Measurement:
  • Demographic Agents: Creating agents based on broad demographic variables (age, job). Prone to stereotyping and oversimplification.
  • Persona Agents: Creating agents based on narrative descriptions. Can be more flavorful but may miss information.
  • Rich Qualitative Data (Interviews):
    • Method: Conducting in-depth interviews (e.g., 2-hour interviews with 1000 Americans using a broad script like "Tell me the story of your life").
    • Digital Twins: Using interview transcripts to create generative agents that serve as "digital twins" of the real individuals.
    • Validation: Having both the real person and their digital twin take the same surveys and experiments (e.g., General Social Survey, Big Five Personality Index, behavioral economic games).
    • Comparison: Measuring how closely the agent's responses replicate the real person's actual behavior and attitudes.

Research Findings on Accuracy

• Normalization: To account for natural variation in human responses over time, results are normalized. A ratio of 1.0 means the agent replicates a person's responses as accurately as that person replicates themselves two weeks later.
• Baseline: Random guessing on a survey yields a replication ratio of about 0.33.
• Persona/Demographic Agents: Achieve a replication ratio of approximately 0.70.
• Full Interview-Based Agents: Achieve a replication ratio of approximately 0.85 on the General Social Survey. This accuracy is maintained even with significantly shorter interviews (e.g., 20% of the original length), suggesting the richness of the data is key.
• Bias Reduction: Rich qualitative data reduces stereotyping and bias compared to using only demographic variables.
• Politics as a Challenge: Politics is the hardest domain to model, with a smaller accuracy gap (around 8%) between the best and worst performing groups. Far-right conservatives were particularly challenging for LLMs, potentially due to value alignment and the subjects' reticence.
• Gender and Race: Showed smaller gaps than expected, with interviews further reducing differences.
• Replicating Scientific Studies:
  • Agents replicated 4 out of 5 published studies from top-tier journals.
  • The one study not replicated by agents was also not replicated by the thousand real people, indicating it was likely flawed research.
  • Simulations showed a strong correlation (0.85-0.9) with effect sizes of pre-registered, non-public study results.

Building Your Own Agent Bank: Advice and Best Practices

• Bad Approach: Defining agents by a single demographic variable (e.g., "be a conservative"). This leads to stereotyping and underestimates variance.
• Not Terrible: Using five or six demographic variables (achieves ~0.70 replication ratio).
• Better: Gathering rich qualitative data, such as in-depth interviews.
• Key Consideration: The data gathered must be relevant to the questions being asked. Interviewing about fashion won't help predict retirement planning unless there's a clear link (e.g., high income).

Risks and Mitigation Strategies

• Errors in Simulation: Simulations can make errors, particularly in quantitative predictions where small percentage differences can be significant.
• The "Ladder of Risk":
  • Possibility Rung (Lowest Risk): Identifying potential outcomes and plausible chains of events. Generally works today. Requires agents to generate plausible scenarios and for users to identify safeguards.
  • Qualitative Rung (Medium Risk): Estimating attitudes and chat-based outcomes. Mostly works today with rich data. Good for rough estimates of reactions to policies or products, but not a replacement for direct engagement.
  • Quantitative Rung (Higher Risk): Measuring quantitative accuracy (e.g., market research surveys). Can have errors. Requires careful validation.
  • Multi-Agent Simulation (Highest Risk): Simulating entire markets or towns. Requires high trust in individual agent accuracy and understanding of emergent properties. High potential for error and difficulty in distinguishing correct from incorrect outcomes.
• Mitigation:
  • In-Domain Data: Ensure agents have relevant data in their memory (e.g., don't use fashion interviews for retirement planning).
  • "Rough-Edged" vs. "Sharp-Edged" Problems: Possibility and qualitative rungs are "rough-edged" – even 80% accuracy is helpful. Quantitative rungs are "sharp-edged" – being wrong can lead to bad decisions.
  • Validation: Validate important questions on a small subsample to check for significant model deviations.

Frontiers and Applications

• "Look Before You Launch" Tools:

  • Application: Preventing policy backfires on online platforms by simulating user reactions to new rules or features.
  • Benefit: Allows for iteration and refinement in simulation before launch, avoiding "dumpster fires."
  • Educational Use: Used in online platform design courses to simulate the impact of trolls and system vulnerabilities.

• Training Soft Skills:

  • Application: Developing conflict negotiation and salary negotiation skills.
  • Method: Generative agents act as "sparring partners" for practice.
  • Experiment: A study showed that participants who underwent simulated conflict negotiation reduced their use of antisocial strategies by two-thirds compared to those who only received a lecture. Simulation aids learning and behavioral change.

• Business Applications:

  • Market Research: Emerging as a powerful tool for market research and understanding consumer behavior.
  • Startup "Simile": A company spun out of Stanford research to commercialize these generative agent technologies.

Conclusion

The development of generative AI agents represents a significant leap towards creating a "what if" machine. By leveraging LLMs and a sophisticated architecture of memory, reflection, and planning, these agents can simulate human behavior with increasing accuracy and believability. While challenges remain, particularly in quantitative predictions and multi-agent systems, the potential applications in decision-making, risk mitigation, and skill development are vast. The key to successful implementation lies in gathering rich, relevant data and understanding the limitations and risks associated with different levels of simulation complexity.