Stanford Seminar - What Brains Forgot, Bodies Remember: Building Intelligence from the Ground Up

Key Concepts:

• Embodied Intelligence: Developing intelligence in machines through sensing, adaptation, and connection, drawing inspiration from developmental cycles in nature.
• Sensing Modalities: Utilizing various sensory inputs like acoustic vibrations, smell (VOC sensors), vision, IMU data, and tactile information for robust environmental perception.
• Adaptation: Enabling machines to adapt to new environments and tasks by developing a "sense of self," capturing body information, and focusing on environmental changes.
• Policy Stitching: Transferring learned skills between robots with different morphologies by decoupling task-related observations from robot-specific observations and enforcing isomorphic transformations in latent feature spaces.
• Behavioral Self vs. Embodied Self: Distinguishing between learning skills and policies (behavioral self) and grounding information in the physical body (embodied self).
• Text-to-Robot: Automating robot design by generating 3D meshes from textual descriptions and co-evolving the robot's body and control policy.
• Symmetric Design: Exploring the benefits of extreme symmetry in robot design for robustness, adaptability, and versatility.
• Human-AI Teaming: Developing platforms and algorithms for effective collaboration between humans and AI agents, leveraging human feedback and theory of mind.
• Implicit Neural Representation: Encoding scene information by querying points in 3D space for properties like distance, color, semantics, and traversability.
• Continuous Traversability Score: Measuring the deviation from a stable pose to assess the traversability of terrain.

1. Sensing: Enabling Machines to Perceive the Environment

• Sonic Sense: A robotic hand uses contact microphones embedded in its fingertips to perceive the environment through vibrations.
  • Example: The robot can identify the number of dice in a container or the amount of water being poured into it by analyzing vibration signals.
  • The system uses T-distributed Stochastic Neighbor Embedding (t-SNE) to reduce the dimensionality of acoustic signals, creating clusters that represent different states (e.g., water levels, number of dice edges).
• Material Composition Detection: The robot taps objects with different fingertips to determine the material composition per pixel.
  • A dataset of 83 objects with labeled material compositions is used.
  • Majority voting in the neighborhood of predictions improves accuracy, leveraging the assumption that material compositions don't change drastically pixel by pixel.
  • Sparse contact points from the object, combined with vibration data, enable 3D reconstruction of the object's shape.
• Smell (VOC Sensing): Using an array of low-cost Volatile Organic Compound (VOC) sensors to detect and identify fungi species.
  • The system addresses the limitations of traditional methods like microscope imaging, DNA sequencing, and e-noses, which are time-consuming and expensive.
  • A customized PCB board with six VOC sensors, humidity, and temperature sensors costs less than $100.
  • An air chamber with six sensor arrays and a robotic arm is used to collect data and train machine learning models.
  • The system can detect fungi species, direction, and distance in about 7 seconds.
  • Example: A robot equipped with the sensor array can autonomously navigate to and identify fungi in a house.
• Multimodal Sensing in Unstructured Environments: Combining multiple sensing modalities (contact microphones, vision, IMU data, LiDAR) for robust perception in environments like forests.
  • Contact microphones are placed on the robot's legs to sense vibrations transmitted through the ground.
  • The system goes beyond semantic understanding to capture the dynamics of the terrain and enable safer and more energy-efficient traversal.
  • A "continuous traversability score" measures the deviation from a stable pose to assess terrain traversability.
  • Implicit neural representation is used to train a model that can answer questions about the environment, such as distance to objects, semantic meaning, and traversability score.
  • Real-world experiments demonstrate the robot's ability to navigate challenging terrains, such as sandy areas and high vegetation.

2. Adaptation: Enabling Machines to Adapt to New Environments and Tasks

• Sense of Self: Developing a "sense of self" that captures body information, allowing machines to adapt to new environments and tasks without relearning their own capabilities.
• Self-Modeling: A robot arm moves randomly while five cameras capture its point cloud.
  • This data is used to train a self-model using implicit neural representation.
  • The model takes joint angles and XYZ location as input and predicts whether the body occupies that space or how far away it is.
  • The robot can generate animations of its body and predict its appearance in different configurations.
  • The self-model enables the robot to detect anomalies and diagnose issues by comparing its predicted appearance with its actual appearance.
• Policy Stitching: Transferring learned skills between robots with different morphologies.
  • The approach involves decoupling task-related observations (e.g., object positions and velocities) from robot-specific observations.
  • A reinforcement learning policy is structured as an encoder-decoder, with the encoder processing task-related observations and the decoder generating actions based on robot-specific observations.
  • Relative repositioning is used to enforce isomorphic transformations in the latent feature spaces of different robots, ensuring that they encode task information in the same coordinate system.
  • Experiments demonstrate that policy stitching enables robots to successfully transfer skills and adapt to new environments, even with different camera configurations.
• Behavioral Self vs. Embodied Self:
  • Behavioral self focuses on learning skills and policies with a fixed body.
  • Embodied self focuses on grounding information in the physical body and understanding the physics of interaction.

3. Design: Automating Robot Design and Exploring Symmetry

• Text-to-Robot: Automating robot design by generating 3D meshes from textual descriptions and co-evolving the robot's body and control policy.
  • The system uses generative models to create 3D meshes based on textual descriptions (e.g., "a robot that looks like a frog but runs as fast as possible").
  • Algorithms are used to determine optimal motor placement and physical parameters.
  • The robot's body and control policy are co-evolved through a loop of design, simulation, and optimization.
  • The resulting robot can be 3D printed and assembled without screws.
  • The approach leverages the strong priors of generative models to create more effective robot designs compared to random combinations of primitive geometries.
• Symmetric Design (Argus Robot): Exploring the benefits of extreme symmetry in robot design.
  • The Argus robot has 20 legs arranged symmetrically around its body, each with an independent motor and a camera.
  • The symmetric design provides robustness, adaptability, and versatility.
  • The robot can walk on rough terrain, carry payloads, and recover from leg failures.
  • The robot can also perform tasks such as climbing between walls and manipulating objects using vision.

4. Connection: Enabling Human-AI Teaming

• Human-AI Teaming: Developing platforms and algorithms for effective collaboration between humans and AI agents.
• Crew Platform: An open-source platform for simulating and experimenting with teams of human and AI agents.
  • The platform allows for real-time eye-tracking and EEG data collection to understand human brain function during collaboration with AI.
• Reinforcement Learning with Human Feedback: Training AI agents to perform tasks based on real-time human feedback.
  • Humans provide feedback (e.g., "good job" or "bad job") to guide the agent's learning process.
  • The agents learn to ground the feedback into individualized controllers.
  • Example: A team of hider robots and seeker robots learns to collaborate based on human coaching.
• Wildfire Response Simulation: Using large language models to simulate and coordinate teams of heterogeneous agents (e.g., bulldozers, drones, firefighters) for wildfire response.

5. Conclusion:

The presentation outlines a comprehensive vision for developing embodied intelligence in machines, emphasizing the importance of sensing, adaptation, and connection. The research explores novel sensing modalities, self-modeling techniques, policy stitching, automated robot design, symmetric design principles, and human-AI teaming. The ultimate goal is to create robots that can perceive the environment, adapt to new situations, and collaborate effectively with humans to solve complex problems and accelerate scientific discovery. The speaker advocates for a full-stack mindset, considering the body, brain, and humans as a closely coupled loop.