Stanford CS25: Transformers United V6 I From Representation Learning to World Modeling

Key Concepts

• JEPA (Joint Embedding Predictive Architecture): A framework for learning world models by predicting future states in a latent (abstract) space rather than pixel space, avoiding the need for decoders.
• World Model: A simulator that predicts future states based on current states and actions, essential for physical AI and robotics.
• Collapse: A failure mode in energy-based models where the model ignores inputs and outputs constant vectors; prevented via contrastive learning or regularization.
• Causal JEPA: An approach focusing on object-centric latent intervention to understand object dynamics and interactions.
• LORE (Latent Object-centric REpresentation) Model: An end-to-end JEPA training method using C-Reg (Isotropic Gaussian Regularizer) to avoid collapse without complex tricks.
• Object-Centric Learning: Representing scenes as sets of objects (slots) rather than raw pixels or patches, allowing for better reasoning about physical interactions.
• C-Reg (Isotropic Gaussian Regularizer): A statistical regularization technique that forces latent embeddings to follow a Gaussian distribution, ensuring informative and non-collapsed representations.

1. Main Topics and Frameworks

JEPA vs. Generative World Models

• Generative Models: Predict pixel-level details, which is computationally expensive and often unnecessary (e.g., modeling every leaf on a tree).
• JEPA: Operates in latent space. It treats prediction as an energy-based model where the goal is to minimize the energy (prediction error) for plausible futures. It avoids the "decoder" bottleneck, focusing only on predictable, meaningful information.

Causal JEPA (Hazel’s Work)

• Goal: Understand object interaction and dynamics.
• Methodology: Uses Slot Attention to bind features to object-specific slots.
• Object Masking: A key training technique where specific object slots are masked. The model must infer the state of the masked object by observing interactions with other objects (e.g., if a monkey is eating, the model infers the banana's state even if the banana is hidden).
• Action Conditioning: Instead of concatenating action embeddings to patches, actions are treated as separate nodes in a graph, improving the model's ability to understand causal influence.

LORE Model (Lucas’s Work)

• Simplicity: A "pure" JEPA implementation requiring only one hyperparameter ($\lambda$).
• Efficiency: 15 million parameters, trainable on a single GPU, and 50x faster than the Dyno world model for planning.
• C-Reg: Uses the Wold Theorem to project high-dimensional embeddings into 1D random directions, optimizing them to be Gaussian to prevent collapse.

2. Real-World Applications and Experiments

• Push T: A robotic manipulation task where the agent must push a T-block to a target. LORE outperformed models using proprioception despite not using it itself.
• Clever: A VQA (Visual Question Answering) dataset used to test counterfactual reasoning (e.g., "What if this object didn't exist?").
• Intuitive Physics: Tested by introducing perturbations (e.g., object teleportation). The model showed high prediction error when physical laws were violated, indicating it had learned the underlying dynamics.

3. Key Arguments and Perspectives

• The Necessity of World Models: Lucas argues that diffusion-based models (like Sora) are not sufficient for physical AI because they are not trained to predict the consequences of actions. True physical intelligence requires a model that understands cause-and-effect.
• System 1 vs. System 2: The speakers suggest a future where agents use "System 1" (fast, reactive policies) for routine tasks and "System 2" (Model Predictive Control/planning) for complex, high-stakes scenarios.
• Hallucination: JEPA models are less prone to the specific types of hallucinations seen in LLMs because they are grounded in physical dynamics and energy-based compatibility checks.

4. Notable Quotes

• "If you want to have physical AI, basically you need world models; you cannot bypass that." — Lucas Amaze
• "The core is masking... if you mask everything, the model doesn't have a shortcut. It needs to infer the other slots to correctly infer the current state." — Hazel (Hi Jang Nam)

5. Synthesis and Conclusion

The lecture highlights a shift in AI research from purely generative, pixel-heavy models toward abstract, object-centric predictive architectures. By utilizing techniques like object masking and statistical regularization (C-Reg), researchers are creating models that are not only more efficient and faster at planning but also possess a deeper, more "human-like" understanding of physical dynamics. The primary research opportunities lie in scaling these models to long-horizon planning, hierarchical reasoning, and real-world, stochastic environments like robotics.