Stanford CS547 HCI Seminar | Autumn 2025 | Building Control and Trust in Human-AI Systems

Key Concepts

• Vibe Coding: A term used to describe AI's ability to generate code or content that aligns with a desired aesthetic or feeling, often without explicit instruction.
• Gemini: A multimodal AI model developed by Google, capable of understanding and operating across different types of information, including text, images, audio, video, and code.
• Visual Metaphor: A design technique where two distinct concepts are visually combined to create a new meaning, often used in advertising and creative content.
• Abstract Design Pattern (Schema): Underlying structural principles or templates that govern creative outputs, such as the three-act structure in storytelling or shape-based integration in visual design.
• Human-AI Collaboration: The synergistic partnership between humans and AI, leveraging their complementary strengths for enhanced creativity, efficiency, and problem-solving.
• Proactive AI: AI systems that can initiate actions and tasks without explicit human prompting, based on learned policies and user goals.
• Control Widgets: Interactive graphical elements that allow users to manipulate and refine AI-generated content, providing granular editorial control.
• User-Authored Policies: Rules and guidelines created by users to constrain and direct the behavior of AI systems, ensuring alignment with human intent and values.
• Shared Problem Representation: A common understanding or framework (like schemas or design patterns) that facilitates effective communication and collaboration between humans and AI.
• Semantic Animation: Animations where elements move and behave in ways that are semantically appropriate to their nature (e.g., a skier moving down a slope).
• Program Repair: The ability of AI systems to automatically detect and fix errors in generated code or content.
• Double Agents: A system that uses AI agents to manage complex tasks like scheduling, guided by user-defined policies.
• Exposure Therapy (for AI Trust): The process of gradually increasing user interaction and exposure to AI systems, leading to increased comfort and trust.
• Transparency: The ability to understand how an AI system arrives at its outputs, crucial for building trust.
• Gatekeeping (in AI Content): Mechanisms to prevent the proliferation of low-quality or unoriginal AI-generated content, such as not monetizing "slop."

Collaboration: Complementary Skills in Design and Creativity

The discussion begins by highlighting the impressive capabilities of AI, exemplified by "vibe coding" and the Gemini model's ability to solve a 3D Rubik's Cube. However, a critical observation is made: the AI's Rubik's Cube solution simply reversed the scrambling steps, demonstrating a lack of genuine problem-solving and an over-reliance on assumptions. This leads to the central argument that AI needs people and cannot be fully trusted as is.

The speaker proposes a framework for building trust in AI, starting with collaboration. The core idea is that humans and AI possess complementary skills:

• Ideation: AI can assist, but human input is needed to avoid blandness.
• Prototyping: AI is significantly faster.
• Testing: Human judgment is essential due to AI's unreliability.

Visual Metaphors and Abstract Design Patterns

A concrete problem is presented: creating creative ads using visual metaphors. The example of The Economist's ad, blending the Christ the Redeemer statue with a symbol of growth, illustrates this. The speaker defines a "good blend" as one where elements are integrated and individually identifiable, leading to the concept of an abstract design pattern. These patterns, like the integration of shapes (circles in the Starbucks example) or thematic connections (Toblerone and fire extinguisher for heat/global warming), operate at a deep structural level, not just surface appearance. This is analogous to the three-act structure in storytelling.

The process for creating such visual metaphors involves:

1. Breaking down concepts: Brainstorming related ideas (e.g., Starbucks: store, Frappuccino, green aprons).
2. Image collection and shape annotation: Gathering images of brainstormed objects and identifying their shapes.
3. Constraint solving: Using the identified design patterns to blend concepts based on shape compatibility.
4. AI-assisted mockup generation: Utilizing AI to visualize the blended concepts.
5. Evaluation and iteration: Refining the designs.

This human-AI workflow, when applied, has been shown to make people 10 times better at creative tasks, enabling them to produce outputs much faster. The key insight is that these abstract design patterns act as a constraint that speeds up the process for both human-AI collaboration and human-human collaboration.

Discovering and Applying Problem Representations (Schemas)

The speaker's team spent five years identifying abstract design patterns for various design challenges, including news TikToks, teaser trailers (now integrated into Adobe products), humor applications, and science explanations on Twitter. The challenge shifted to finding these problem representations, which are also called schemas or design patterns. This process involved extensive hypothesis testing and staring at examples.

A system called Schemax was developed to discover and apply these schemas. The workflow involves:

1. Data Input and Clustering: Providing a dataset (e.g., shapes) and using AI for clustering (e.g., squares, triangles, circles).
2. Schema Guessing: AI (or a human) proposes a schema for a cluster (e.g., "three sides" for triangles).
3. Output Generation and Testing: Generating outputs based on the proposed schema and comparing them to other examples in the cluster.
4. Schema Refinement: Identifying differences and using AI to improve the schema. This loop repeats.

The system performs generate-and-test hypothesis testing. Humans are involved in the loop for validation. The output is a schema that fits the data for each cluster.

A video demonstration of Schemax for writing HCI paper abstracts illustrates the process:

• Goal: Write HCI paper abstracts.
• Data: 15 HCI paper abstracts paired with titles.
• Clustering: AI identifies clusters like "empirical studies," "theoretical contributions," and "system design and evaluation."
• Schema Induction: For a selected cluster (e.g., empirical studies), AI infers dimensions like "motivation," "problem," "method," "findings," and "implications."
• Attribute Inference: AI identifies overall attributes (e.g., word count, narrative structure) and dimension-specific attributes (e.g., qualitative insights, quantitative evidence).
• Human Review: The user checks the consistency and accuracy of AI's findings.
• Schema Application: The system uses the schema and paper titles to generate draft abstracts.
• Contrastive Refinement: The AI compares generated abstracts to actual examples and iteratively refines the schema.

The benefit of this schema-driven approach is that when an LLM is prompted with a schema, it performs significantly better than without it. Crucially, participation in schema creation leads to understanding and belief in the AI's output, fostering a shared representation.

The application of these schemas extends to practical tools, such as an app developed with Adobe that uses schemas to adapt ad campaigns. For example, a Burger King ad's structure (abstract stripes, object morph, deconstruction, typography reassemble) can be applied to an environmental campaign, reusing elements like color palettes and objects, and even identifying rhythmic dimensions synchronized to music. AI can then generate video prompts for models like V3.

Control: Semantic Animations and Editorial Interfaces

The second major topic is control, focusing on AI-generated animations. The goal is to create semantic animations where elements behave naturally (e.g., a skier skiing, a cat curling up). This is difficult with traditional templated animation code.

Logo Motion is a system that generates candidate animations, fixes bugs, and implements them in code (e.g., anime.js). It provides editorial control through a timeline, version history, and a layer panel.

The process involves:

1. Asset Input and Interpretation: Providing assets (images) and having AI identify them (e.g., "skier," "mountains").
2. HTML/JavaScript Representation: Translating semantic information into code.
3. Design Concept Generation: AI proposes animation concepts with hero moments and sequencing.
4. Full Implementation: Generating the animation code.
5. Self-Debugging: The AI plays the animation, checks for errors (e.g., skier too low), and precisely fixes only the erroneous elements by adjusting code. This is a significant improvement over systems that rewrite the entire code.
6. Editorial Control: Users can then use editing widgets to further refine the animation without affecting other parts. These widgets are connected to the backend code.

The editing interface allows users to:

• Explore an exploration tree of animation options.
• Use a narrative timeline to understand element behavior.
• Write prompts to modify animations (e.g., "Can it do a cool trick?").
• Synchronize timings across elements.
• Delete moments.
• Regroup layers.
• Use quick actions like "subtle" or "emphasis" to modulate animation intensity.
• Drag and drop blocks for absolute control over timing.

The system's program repair has a 95% solve rate, ensuring that AI fixes its own mistakes. The editing agents help users explore and craft better iterations, injecting a "human touch" or "pizzazz" that AI alone might miss. The ultimate goal is to create animations that evoke joy.

A generalized version of this system can analyze any code, identify states and transitions, and create editing interfaces for them, not just animations.

Trust: Proactive AI and User-Authored Policies

The third topic is trust, exemplified by the Double Agents project, which aims to automate complex scheduling tasks. The problem is the time-consuming and often frustrating back-and-forth of email communication for organizing events.

The core principle for building trust in this context is constraining AI actions based on user-authored policies. These policies are tested rigorously through multi-agent simulations.

The Double Agents workflow:

1. Goal Specification: User defines the goal (e.g., organize a seminar with four speakers and four slots).
2. Policy Definition: User creates policies (e.g., "ask all speakers for availability across all slots," "follow up every two working days").
3. Simulation: The system simulates interactions with personas (e.g., busy professor, eager student) to test policies.
4. Planning: Based on policies, the AI generates a plan for actions.
5. Action Generation: The AI drafts concrete actions (e.g., draft and send emails).
6. User Review and Approval: The user reviews and approves AI-generated emails and plans.
7. Edge Case Handling: When an unexpected situation arises (e.g., a speaker requests Zoom), the AI flags it as an edge case, prompting human intervention.
8. Policy Inference: Over time, the AI can infer new policies by observing user actions in response to edge cases.

User studies revealed that participants were initially hesitant to trust AI with real emails to famous faculty. However, exposure therapy, visualization of AI's reasoning (through policies), and rehearsal on realistic scenarios (simulations) increased comfort and trust. This leads to proactive AI, where the system can act without constant prompting.

Transparency is identified as a key component of trust, similar to human-to-human trust. The ability to see the steps the AI is making, especially when it makes mistakes, is crucial. While some argue that only the result matters, the speaker counters that AI's frequent errors necessitate transparency in its process.

The system aims to minimize the "GPT tone" in AI-generated emails by learning from user edits and preferences. Over time, users become more comfortable accepting AI-generated actions. The ideal future state involves automated interfaces where users are only alerted to critical flags (e.g., a speaker requesting flight reimbursement).

The system is being extended to manage product releases and unblock employees, mirroring the manager's role in resolving bottlenecks.

Synthesis and Conclusion

The presentation argues that while AI is powerful, it is far from perfect and often makes assumptions that can be misleading. Therefore, AI needs people. The key to effective human-AI collaboration lies in:

1. Shared Representations: Discovering and utilizing abstract design patterns or schemas that provide a common framework for both humans and AI. This accelerates creativity and problem-solving.
2. Control: Providing users with visual widgets and editing interfaces that allow for granular manipulation and refinement of AI-generated content, ensuring human oversight and creative direction.
3. Trust: Building trust through user-authored policies that constrain AI behavior and are rigorously tested. Transparency in AI's reasoning and error handling, along with proactive flagging of edge cases, is essential.

The speaker emphasizes that the goal is not to replace human creativity but to augment it. Even in creative fields, underlying structures (schemas) exist, and artists often leverage these while injecting their unique perspectives. AI can act as a skilled execution agent, but the original ideas and concepts still stem from human insight. The challenge of preventing AI from producing generic "slop" is addressed through mechanisms like not monetizing low-quality content, thereby preserving signals of value. Ultimately, the aim is to create AI systems that are not just functional but also capable of producing outputs that evoke emotion and joy, driven by a deep understanding of human values and intentions.