Stanford Seminar - Robot Learning Without Action Chunking

Key Concepts

• Action Chunking: Grouping multiple actions into a single decision.
• Temporal Dependencies: The influence of past actions and internal states on future actions.
• Reactivity: The ability of a policy to respond to unexpected changes in the environment.
• Bidirectional Decoding: A test-time algorithm that combines action chunking with guided resampling to balance consistency and reactivity.
• Past Token Prediction (PTP): An auxiliary training task that regularizes a policy to preserve critical information about past actions.
• Multi-Stage Training: A training recipe that pre-trains a visual encoder in a short-context setting and then trains the action head in a long-context setting.

Action Chunking: Strengths and Limitations

• Definition: Action chunking involves grouping multiple actions into a single decision, predicting a sequence of actions simultaneously, and executing them as a chunk.
• Benefits:
  • Captures temporal dependencies in human demonstrations, where internal states and preferences influence multiple actions in a row.
  • Operates as a long-context policy by considering information across many time steps.
• Limitations:
  • Reduces the frequency of observing the environment, hindering reactivity to unexpected events.
  • Creates a trade-off between modeling temporal dependencies and maintaining reactivity, especially in stochastic and uncertain environments.
• Examples:
  • Long chunk size leads to smooth behavior but poor reactivity to moving objects.
  • Short chunk size improves reactivity but can cause oscillations between different strategies.
• Conflicting Observations:
  • Earlier work (ACT, Diffusion Policy) argued that action chunking is critical for strong performance, requiring chunks longer than 1 second.
  • Other papers showed that action chunking can be detrimental in certain situations.

Bidirectional Decoding: Combining Consistency and Reactivity

• Core Idea: Combines the benefits of long-chunk and short-chunk policies at test time through a decoding algorithm.
• Process:
  1. At time step t, a chunk of actions is predicted.
  2. Only the first action (or a small part of the chunk) is executed.
  3. The remaining actions are stored in memory.
  4. At time step t+1, a batch of candidate chunk predictions is generated.
  5. The optimal candidate is selected based on two criteria:
     • Consistency with Prior: Measured by the distance between the candidate and the prior prediction (in data or latent space).
     • Forward Contrast: Maximizes the likelihood of the future plan by selecting candidates close to positive samples (from the best checkpoint) and far from negative samples (from suboptimal checkpoints).
• Benefits:
  • Maintains long-term consistency while adapting to new observations.
  • Provides robust and substantial gains compared to baselines like random sampling, warm start, and temporal ensembling (EMA).
  • Benefits from additional compute at test time (more samples lead to better solutions).
• Limitations:
  • May become inefficient as the number of latent strategies grows with increasing demonstration data.
• Real-World Application: Demonstrated on dynamic manipulation tasks (pushing a T-block, catching a cup) with perturbed object locations.

Past Token Prediction: Learning History Representations

• Motivation: To address the limitations of bidirectional decoding when dealing with a large variety of latent strategies.
• Core Idea: Train the policy to predict not only future actions but also past actions, explicitly regularizing it to preserve critical information about the past.
• Challenges in Building Long-Context Policies:
  • Performance drop due to the use of spurious features.
  • Slow training process due to increased memory requirements.
• Existing Approaches and Their Limitations:
  • Truncating history (used in RT-X, PI Zero): Throws away potentially critical information.
  • Compressing history into few frames: Hand-engineered representations that may discard important information.
• Multi-Stage Training Recipe:
  1. Pre-train the visual encoder in a short-context setting.
  2. Pre-process and cache the visual representations.
  3. Train the downstream action head in a long-context setting using the cached embeddings.
• Benefits:
  • Greatly improves performance on tasks requiring memory from the history.
  • Significantly improves learning efficiency (achieves similar performance with 10% or less computational time).
• Real-World Application: Demonstrated on tasks like moving beans to a bowl and replacing objects.

Synthesis/Conclusion

The presentation addresses the trade-off between long-term consistency and short-term reactivity in robot learning from human demonstrations, particularly in the context of action chunking. Bidirectional decoding offers a test-time solution to balance these factors by combining action chunking with guided resampling. Past Token Prediction provides a training-level solution by enabling the policy to learn history representations directly from demonstration data, improving performance and learning efficiency. These methods contribute to building more robust and adaptable robot policies capable of handling complex tasks in dynamic environments.