Stanford Seminar - Beyond Words: Affordances of Embodiment in Automated Health Counselors

Key Concepts

• Non-verbal behavior in healthcare: How providers use their bodies (posture, gestures, facial expressions, proxemics) to communicate with patients.
• Conversational function vs. conversational behavior: Distinguishing between observable actions (e.g., eyebrow raise) and their intended meaning (e.g., emphasis, surprise).
• Grounding: The process of establishing mutual understanding in a conversation.
• Dyus: Using language or non-verbal cues to reference something in the context (e.g., pointing, gaze).
• Metaphoric gestures: Using hand gestures to represent abstract concepts through spatial mappings.
• Relational communication: Cues used to establish or modify interpersonal relationships (e.g., trust, working alliance).
• Health literacy: The ability to find, process, understand, and act on medical information.

Body Language in Health Counseling

The speaker, a native Californian working in Boston, discusses the importance of non-verbal behavior in healthcare, particularly in automated conversational systems like chatbots. While many chatbot interventions focus on text-based interactions, they often lack the crucial element of a "body" – the non-verbal cues that healthcare providers use to build rapport, convey empathy, and ensure understanding.

Personal Anecdotes

• Negative Example (Colonoscopy Consent): The speaker recounts a negative experience where a nurse provided a consent form with clear frustration and upset conveyed through proxemics, posture, and facial expressions, highlighting the importance of attitude in healthcare interactions.
• Positive Example (Pediatrician Visit): A positive experience involved a pediatrician who adjusted her stool to be at eye level with the speaker's daughter, using a warm expression and compassionate tone to ease the child's anxiety.

Importance of Empathy

Studies show that empathy conveyed by physicians leads to improved patient satisfaction and lower rates of malpractice suits. However, non-verbal behavior goes beyond gross displays of "body language" and includes more subtle cues.

Simulating Non-Verbal Behavior

The speaker's lab focuses on building simulations of verbal and non-verbal interactions for patient-facing health interventions.

Hospital Discharge Nurse Example

• Study: Video recording a hospital discharge nurse explaining instructions to a mock patient.
• Analysis: Studying hand gestures (referring to objects in a booklet), gaze (coordination with hand gestures, turn-taking), and coordination with verbal explanations.
• Simulation: Building a virtual hospital discharge nurse that walks patients through a booklet using similar hand gestures. This was tested in a clinical trial at Boston Medical Center with a booklet designed for patients with low health literacy.

Genetics Counselor Example

• Study: Video recording a genetics counselor conducting an initial counseling session with a mock patient.
• Analysis: Observing attention-checking behaviors, grounding cues, representational gestures (passage of time), facial displays (eyebrow raises for emphasis, disgust), metaphoric gestures, and dyadic gestures (pointing at a pedigree chart).
• Simulation: Building a system called Pearllo, currently in trial at LA General Hospital and University of Rochester Medical Center, to promote germline genetic testing by cancer patients.

Conversational Functions and Non-Verbal Behavior

The speaker emphasizes the distinction between conversational behavior (observable actions) and conversational function (intended meaning). A single behavior can serve multiple functions, and a single function can be achieved through various behaviors.

Grounding

• Definition: The process of negotiating mutual understanding in a conversation.
• Example: A patient understanding that "How are you?" is a bid for ritual chat, not an inquiry into their cancer status.
• Evidence of Understanding: Continued attention, head nodding, backchannel cues, explicit repetitions, demonstrations.
• Application: A project funded by NCI to collect patient-reported outcomes (PROs) at home, focusing on pain.
• System: A furHat robot that captures speech and facial expressions, using them to generate empathic utterances and non-verbal behaviors (head nods, facial displays).
• Experiment: Comparing the empathic robot to a robot that only provides backchannel acknowledgement.
• Results: The empathic robot led to significantly higher ratings on empathic listening, emotional intelligence, and working alliance.

Dyus

• Definition: Using language or non-verbal behavior to reference something in the context.
• Examples: Pointing, referencing current time/place/relationship status.
• Application 1: Shared decision-making conversation around birth control for African-American women.
• System: A virtual agent that uses hand gestures to explain a chart comparing different birth control methods.
• Results: Significantly better at increasing the stage of change for preconception preparation behaviors compared to a non-intervention control group.
• Application 2: Using a humanoid robot (Pepper) to influence food choices in a food pantry setting.
• Experiment: Manipulating proxemics (robot moving into the user's decision space) and persuasive language.
• Results: Persuasion was more effective than none, and moving into the user's location was even more effective in improving the nutritional quality of meals assembled.
• Application 3: Using gaze in multi-party counseling conversations.
• Context: Couples relationship education, teaching communication skills like active listening.
• System: A furHat robot that turns and looks at the individual it is addressing.
• Results: High compliance with robot instructions, comfort with the situation, and higher ratings of partner's empathic listening after interacting with the robot compared to text-based information.

Conveyance of Meaning (Propositional Communication)

• Channel: Hand gestures, particularly metaphoric gestures.
• Example: Genetics counselor using a "light switch" gesture to explain that cancer is not an episodic event.
• Experiment: Investigating if Pepper can use hand gestures to convey complex concepts (e.g., relative risk using a risk ladder).
• Results: People could glean information from Pepper's gestures, with higher hand positions associated with happier emotions and greater quantities.

Conveyance of Emotion

• Channels: Facial displays (concern for empathy), physical touch.
• Application: Exploring the use of touch in conjunction with automated counseling.
• System: A "Touchbot" with a haptic glove that squeezes the participant's hand.
• Experiment: Investigating patterns and pressures of touch that convey different emotions.
• Results: People comfortable with touch had a positive reaction, while those uncomfortable had a negative reaction.

Relational Communication

• Definition: Cues to mark or move a relationship along a trajectory of increased closeness, trust, or working alliance.
• Application: Using VR to explore the impact of proxemics on trust-building in health counseling.
• System: A VR character that incrementally moves closer to the user during a trust-building conversation.
• Results: Increased motivation to change behavior, increased self-efficacy, and closer stopping distance in a subsequent interaction.

When to Embody Chatbots

The speaker discusses when it is advantageous to give chatbots a body.

Sources of Evidence

• Smartphone Interactions: For brief transactions, users prefer standard GUIs. For narrative tasks, preferences are split. Strong idiosyncratic preferences exist.
• In-Group Affiliation: Embodiment can be used to mark in-group affiliation. Culturally tailored agents are preferred by specific communities.
• Disadvantaged Populations (Low Health Literacy):
  • Patients with low health literacy often struggle with text and conventional user interfaces.
  • Virtual hospital discharge nurse study: More patients preferred the conversational agent over doctors/nurses.
  • Patients with low health literacy were significantly more satisfied with the conversational agent.
  • Head-to-head comparisons: Conversational agents outperformed conventional UIs in clinical trial search and family health history collection, especially for those with low health literacy.

Dialogue Engines and Language Models

The speaker notes that their systems currently use deterministic rule-based dialogue flow with template-based text generation for safety and validation reasons. However, they are increasingly using large language models to help design these systems and explore hybrid architectures.

LLMs for System Design

• Developing a tool that uses LLMs to divide patient education materials into a series of conversations.
• LLMs incrementally design each conversation with a dialogue flow-like formalism.
• Clinicians edit and validate the end product before deployment.

Conclusion

The speaker concludes by reiterating the goal of capturing the essence of compassionate, empathetic interactions between providers and patients to maximize health outcomes and patient satisfaction. The presentation highlights the importance of non-verbal behavior in healthcare and explores various ways to simulate these cues in automated conversational systems, particularly for disadvantaged populations with low health literacy.