Stanford Robotics Seminar ENGR319 | Spring 2026 | Leveraging Geometry in Robot Learning

Key Concepts

• Equivariance: A property where a transformation in the input (e.g., rotation or translation) results in a corresponding, predictable transformation in the output.
• Model-Based vs. Generalist Models: The historical shift from hand-coded geometric models (high precision, low data, rigid) to large-scale generalist models (high data, flexible, "disembodied" reasoning).
• Diffusion Policy: A generative approach to policy learning that models the action distribution as a flow field.
• Geometric Transformer Attention (GTA): An attention mechanism that incorporates reference frames into the computation to maintain spatial awareness.
• Data Efficiency: The ability to achieve high success rates with significantly fewer demonstrations by embedding physical symmetries into the model architecture.
• MimicGen: A benchmark suite used to evaluate robotic manipulation tasks across varying levels of complexity.

---

1. The Evolution of Robotics: Bridging the Gap

Rob Platt argues that robotics has moved from hand-structured geometric models (which fail when reality deviates from the model) to large-scale generalist models (which are data-hungry and often "disembodied," losing geometric context during self-attention). The core research question is: Can we create models that leverage geometry and physics while retaining the flexibility of machine learning?

2. Methodologies and Frameworks

Platt presents four specific approaches to encoding geometric observations to improve policy learning:

• Equivariant Diffusion Policy (EDP):
  • Mechanism: Encodes the world as a point cloud and reasons using finite subgroups of SE(3) (Special Euclidean group).
  • Physics Integration: Incorporates Emmy Noether’s principle, which links physical symmetries (translation/rotation) to conservation laws (momentum/angular momentum).
  • Result: Achieves a 10x improvement in data efficiency compared to standard diffusion policies, performing better with 100 demos than baselines do with 1,000.
• Image-to-Sphere Embedding:
  • Mechanism: Projects RGB image patches onto a 2-sphere to facilitate SO(3) rotations. It uses spherical harmonics (Fourier basis for the sphere) to perform convolutions in the Fourier domain.
  • Application: Enables the use of RGB images (specifically eye-in-hand cameras) while maintaining rotation equivariance.
• Raven (3D Ray Embedding):
  • Mechanism: Encodes image patches as 3D rays pointing from the camera origin.
  • Geometric Transformer Attention (GTA): Transforms queries, keys, and values into a common reference frame before performing attention, ensuring the model understands the spatial relationship between different image patches.
• Pix-to-Act:
  • Mechanism: Infers keypoint trajectories in the image plane from multiple cameras and triangulates them into 3D space.
  • Data Augmentation: Uses "visual axis rotation" (simulated spinning of cameras) to force the model to ignore global context and focus on local, task-relevant structures.

3. Key Arguments and Evidence

• The "Pyrrhic Victory" of Big Data: Platt warns that relying solely on massive datasets is a "Pyrrhic victory"—it works, but the supply lines (data collection) are unsustainable.
• Shifting the Scaling Law: The primary goal of incorporating geometry is to shift the "test loss vs. data size" power law curve to the left, meaning the model achieves the same performance with significantly less data.
• Bias-Variance Trade-off: By biasing the model toward physical symmetries, the model spends less "learning capacity" on figuring out basic geometry and more on task-specific nuances.

4. Notable Quotes

• "A key failure mode for robotics until about 10 years ago was a robot failing to pick something up because it misestimated the location of something in the world."
• "The last thing that these models know about the geometric structure of the environment is a position encoding... and those were basically obliterated during self-attention."
• "We want to shift the scaling curve to the left... by making smarter models."

5. Synthesis and Conclusion

The research demonstrates that modern robotics does not need to choose between rigid geometric models and "black-box" generalist models. By embedding equivariance and geometric awareness (via point clouds, rays, or spherical projections) into the architecture of diffusion transformers, researchers can create models that are significantly more data-efficient. These models treat physical constraints not as limitations, but as inductive biases that allow the robot to generalize better across poses and viewpoints, ultimately making high-precision tasks (like coffee making or object insertion) achievable with a fraction of the data required by standard large-scale models.