Stanford Seminar - Generalization through Task Representations with Foundation Models

Key Concepts

• Generalization across environments and tasks
• Task representation
• Foundation models (LLMs, VLMs)
• 3D value maps
• Relational keypoint constraints (ReCaP)
• Affordances
• Imitation learning

Natural Language as Task Representation

• Main Idea: Explores using large language models (LLMs) to represent tasks for robot planning.
• Details:
  • Investigated in the Virtual Home environment (simulated household).
  • Goal: Control an embodied agent to complete tasks using a pre-trained LLM in a zero-shot manner.
  • Early results with GT3 showed promise in decomposing tasks into steps.
  • Google built upon this work by introducing a value function for each skill, combining LLM knowledge with skill-specific values for multi-step task planning.
• Limitation: Robotic tasks are spatial, while natural language is textual. Learning spatial tasks like folding a shirt is much faster through video than through textual instructions.

3D Value Maps for Grounding Tasks in Space

• Main Idea: Represent tasks using 3D value maps generated from language instructions.
• Process:
  1. Given a language instruction (e.g., "open the top drawer"), a foundation model generates code to construct a 3D voxel grid representing the robot's workspace.
  2. The LLM detects objects (e.g., handles, vases) using vision-language models.
  3. Positive values are assigned to locations relevant to the task (e.g., the top handle), and negative values to locations to avoid (e.g., the vase).
  4. The 3D value map is smoothed to propagate values across the workspace.
  5. Motion planning is used to generate robot actions based on the 3D value map.
• Example: Opening the top drawer involves detecting handles, sorting them by height, assigning positive values to the top handle location, and avoiding a vase by assigning negative values to its location.
• Results: Demonstrated on various tasks without task-specific data or demonstrations, achieving 6-DoF control at 10 Hz.

Relational Keypoint Constraints (ReCaP) for Spatial and Temporal Composition

• Main Idea: Represent tasks as a sequence of relational keypoint constraints to capture both spatial and temporal dependencies.
• Process:
  1. Large vision models identify keypoints in the scene.
  2. GPT-4 generates Python functions that define constraints based on the keypoints and task instructions.
  3. Each constraint maps a set of keypoints to a numerical cost.
  4. A task is represented as a series of these constraints (ReCaP functions).
  5. Constraints are of two types:
     • Subgoal constraints: Define what the robot should achieve at the end of a stage.
     • Path constraints: Define constraints that the robot should maintain during a stage.
  6. A constraint optimization solver uses these constraints to generate robot actions.
  7. If a path constraint is violated, the system backtracks to a previous stage.
• Example: Pouring tea involves stages like grasping the handle, keeping the teapot upright, aligning the spout to the cup, and pouring. Constraints are defined for each stage, such as minimizing the distance between the robot's gripper and the teapot handle.
• Results: Enables robots to replan actions in closed-loop at 10 Hz, adapting to disturbances. Demonstrated on tasks like recycling cans, placing shoes, folding blankets, and stowing books.

Affordances as Task Representation

• Main Idea: Represent tasks by affordances, which are action possibilities afforded by the environment.
• Process:
  1. Collect data autonomously by rendering 3D object models from multiple viewpoints.
  2. Compute DinoV2 features for each rendered image and fuse them into a 3D feature field on the object point cloud.
  3. Perform clustering to generate semantic regions for each object.
  4. Use a VLM to brainstorm potential tasks and match them with associated regions. For example, the rim of a cup is associated with drinking.
  5. Compute feature similarity against the 3D feature field to generate continuous annotations.
  6. Train an affordance model on this data to predict affordances for different objects and tasks.
• Example: Identifying the seat of a chair as having high "sitting" affordance, or the ear cups of headphones as having high "listening" affordance.
• Results: A single model can generalize to unseen poses, object instances, object categories, and task instructions. Demonstrated on tasks like watering plants and throwing trash. The model runs at approximately 200 Hz on a modern GPU.

Task Representation with Demonstrations

• Main Idea: Use demonstrations to create a fine-grained and composable task representation.
• Process:
  1. Given tightly operated interaction trajectories (demonstrations), use foundation models to decompose them in space and time.
  2. Along the time dimension, decompose the demonstration into sub-tasks (e.g., grasping a green book, adjusting a pink book, placing the green book on the shelf).
  3. Along the space dimension, segment the relevant objects.
  4. Train imitation learning policies on this data.
• Results: Enables generalization to tasks with different object heights, geometric constraints, and task horizons.

Closing Notes and High-Level View

• Challenge: Despite progress in scaling robotic data collection, achieving truly autonomous robot butlers remains a challenge.
• Proposed Approach: Leverage internet-scale data and foundation models to provide common sense and task-specific knowledge.
• Key Components:
  • Symbolic Intelligence: LLMs and VLMs provide semantic understanding of tasks.
  • Task and World Model: A spatial (3D) model of the environment that captures how the world works.
• Vision: Combine task-specific knowledge from foundation models with a spatial task and world model to enable robots to operate autonomously and generalize across tasks and environments.

Q&A Highlights

• Motion Planning: The speaker views motion planning as one way to generate actions, alongside prediction. Planning relies on a model of the environment and an objective. Simplified models (e.g., rigid attachment assumption) can be used, but a general-purpose world model would be ideal.
• Object Segmentation: The speaker acknowledges the importance of a feedback loop between the VLM and the segmentation pipeline to determine the optimal number of segments. Over-segmentation is currently used, and the VLM selects the relevant segments. The mean shift clustering algorithm is used for segmentation, with a threshold set empirically in the feature space.