Stanford Seminar - Evaluating and Improving Steerability of Generalist Robot Policies

Key Concepts

• Generalist Manipulation Policies: Robot policies capable of performing a wide range of tasks with various objects in diverse environments, controllable via natural language.
• Generalization: The ability of a policy to perform well in novel situations, categorized by robustness to perturbations in visual, semantic, and behavioral modalities.
• Steerability: The ability of a language-conditioned policy to accurately follow complex and fine-grained language instructions.
• Vision Language Action (VLA) Models: Models that take visual observations (images) and language instructions as input and output actions for a robot to execute.
• StarGen Taxonomy: A framework for categorizing and measuring generalization in robot policies based on perturbations in visual, semantic, and behavioral modalities.
• Posterior Collapse: A phenomenon where a model ignores language conditioning signals because the task can be solved without attending to the language.
• Counterfactual Re-labeling: A technique to augment data sets by generating new actions and behaviors using a forward model and labeling them with new language.

Evaluating Generalization

Types of Generalization

The speaker emphasizes the importance of systematically categorizing generalization types to predict model properties. Generalization is viewed through the lens of the three modalities of a VLA policy:

• Visual: Generalization to changes in visual observations (e.g., lighting, background, object appearance). Example: Putting a carrot on a plate in a red-colored sink instead of a standard one.
• Semantic: Generalization to variations in language instructions (e.g., synonyms, rephrasing). Example: Saying "put the orange object on the plate" instead of "put the carrot on the plate."
• Behavioral: Generalization to changes in required actions (e.g., different object positions, obstacles). Example: Putting the carrot on the plate when it starts in a different position in the sink.

StarGen Taxonomy

The StarGen taxonomy is introduced as a structured way to measure robustness to perturbations along these modalities. It involves:

1. Base Task: A starting task (e.g., "carrot on a plate").
2. Factor: A specific change to the base task (e.g., changing lighting conditions).
3. Axis: A grouping of similar factors (e.g., image augmentations like lighting changes, cropping, skewing).
4. Category: A broad grouping of related axes (e.g., visual category encompassing all axes related to image input).

The taxonomy helps quantify how well a policy generalizes to perturbed settings and assesses the visual, semantic, and behavioral homogeneity required for the policy to function.

Experimental Results

The StarGen taxonomy was applied to evaluate open-source VLA models on the Bridge dataset:

• Models: OpenVLA, Mini (smaller OpenVLA), and Pi Zero (more powerful VLM backbone).
• Findings:
  • Better VLM backbones (like in Pi Zero) improve performance across the board.
  • Models are generally good at visual generalization (new objects, distractors).
  • Models struggle with viewpoint changes.
  • Semantic generalization is surprisingly poor, indicating a lack of true language grounding.
  • Behavioral generalization often results in near-zero performance, suggesting fundamental issues.
  • Data diversity is crucial; training on a larger, more diverse dataset (OXE) significantly improves performance.
  • Co-training with VQA data primarily helps in the visual domain.

Key Takeaways

• Architectural convergence is still far off; continuous improvements in VLM architectures are needed.
• Data remains a critical factor influencing performance.
• Breaking down generalization performance is essential for understanding model strengths and weaknesses.

Improving Steerability

The Challenge of Steerability

Steerability is defined as the ability of a language-conditioned policy to follow complex and fine-grained language instructions. The speaker argues that steerability is orthogonal to generalization; a general policy may still be unusable if it cannot be steered effectively.

The Labeling Problem

The core hypothesis is that poor steerability stems from a labeling problem in existing datasets. Many datasets lack sufficient language grounding signal, leading to posterior collapse where the model ignores language conditioning because it can perform the task without it.

Relabeling for Better Steerability

Two approaches to improve steerability are presented:

1. Relabeling Existing Datasets:

   • Leverage the existing behavioral diversity in datasets by rephrasing labels to capture nuances in actions.
   • Example: Instead of "pick up the object," use labels like "grasp this particular object in a particular way" or "reorient the object so it matches a certain pose."
   • This can be achieved by clustering grasp poses and associating them with language labels (e.g., "top-down grasp," "diagonal grasp").
   • Pairing these relabeled policies with VLMs allows for more complex task execution by breaking down tasks into parameterized primitives.

2. Counterfactual Re-labeling:

   • Address the lack of behavioral diversity by generating counterfactual actions (actions not in the original dataset) using a forward model.
   • This widens the data distribution and allows for learning from alternative outcomes.
   • Example: In navigation, sample collision-free actions that the robot could have taken but didn't, and label them with corresponding language instructions.
   • This approach requires a suitable forward model, which is easier to build in some domains (e.g., 2D navigation).

Experimental Results

• Relabeling datasets with more specific language instructions improves the ability of policies to follow those instructions.
• Counterfactual re-labeling in navigation tasks allows the robot to perform more complex behaviors, such as moving towards specific objects or avoiding obstacles.

Key Takeaways

• Steerability is crucial for making generalist policies usable.
• Improving language grounding in datasets is key to enhancing steerability.
• Relabeling and counterfactual re-labeling are effective techniques for achieving this.

Scaling Up with Gemini Robotics

Gemini Robotics Project

The Gemini Robotics project at DeepMind aimed to build a VLA from scratch, leveraging the Gemini 2.0 VLM. The process involved:

1. Adding Robotic Capabilities to the VLM: Incorporating spatial reasoning, embodied reasoning, object detection, and grasp detection.
2. Adapting the VLM into a VLA: Fine-tuning the VLM with diverse robotics data containing language labels.

Gemini Robotics VLA

• Trained on thousands of hours of Aloha robot data, encompassing diverse tasks, objects, and manipulation skills.
• Hybrid architecture with a cloud-based VLM backbone and a local action decoder for real-time control.
• Capable of performing a wide range of tasks autonomously, including folding laundry and packing bags.

Generalization and Steerability Evaluation

• Visual Generalization: The Gemini Robotics model demonstrates strong visual generalization, maintaining performance even with distractor objects, different backgrounds, and varying lighting conditions.
• Semantic Generalization: While still performing well, the model shows a performance drop when faced with misspellings, different languages, rephrasing, or increased descriptiveness in instructions.
• Behavioral Generalization: Performance suffers significantly when dealing with completely new object instances, highlighting the challenge of generalizing to novel behaviors.
• Steerability: The model retains a high degree of steerability, attributed to the use of re-labeling and language grounding techniques during training.

Key Takeaways

• Scaling up VLAs requires a principled approach to pre-training and fine-tuning.
• Visual generalization is relatively well-addressed, but semantic and behavioral generalization remain challenging.
• Re-labeling and language grounding are crucial for achieving high steerability in large-scale VLA models.