Stanford CS25: Transformers United V6 I From Language Models to Native Multimodal Intelligence

Key Concepts

• Native Multimodal Language Models: AI systems designed to process and generate multiple modalities (text, image, audio, video) natively within a single transformer architecture.
• Tokenization: The process of converting diverse data types (images, audio) into a sequence of tokens (discrete or dense vectors) that a transformer can process.
• Omni-models: Models capable of both multimodal input and multimodal output (e.g., GPT-4o).
• Patchify: A technique to divide images into small, standardized units (e.g., 16x16 pixels) for encoding.
• Chameleon: A model family that discretizes all modalities into tokens using VQ-VAE, allowing for interleaved text-image generation.
• Transfusion: An architecture that combines auto-regressive language modeling (for text) with diffusion-based generation (for images) in a single transformer.
• Mixture of Transformers (MOT): An architecture using modality-specific parameters (QKV matrices and feed-forward layers) to handle different data densities efficiently.
• Positive Transfer: The phenomenon where learning in one domain (e.g., language reasoning) improves performance in another (e.g., image generation).

---

1. Building Native Multimodal Models

The core philosophy of modern multimodal AI is to treat all data as a sequence of tokens. By "patchifying" images or transforming audio waveforms into vector representations, these signals become compatible with the transformer architecture.

• Tokenization Strategy: Images are broken into patches, encoded, and sequentialized. Videos are treated as sequences of images.
• Input vs. Output:
  • Multimodal Input/Text Output: Models like Gemini or Qwen focus on understanding.
  • Omni-models: Models like GPT-4o generate both text and non-text modalities.

2. Architectural Frameworks

• Chameleon (Discrete Approach): Uses VQ-VAE to map image patches to a learned codebook of discrete indices. This allows the model to treat images like "visual words," enabling interleaved text-image generation. Limitation: Significant information loss during discretization and high token budget requirements.
• Transfusion (Hybrid Approach): Unifies auto-regressive modeling (for text) and diffusion (for images). It uses causal attention for text and bidirectional attention for images. Benefit: Higher image quality and better token efficiency than discrete methods.
• Mixture of Transformers (MOT): Addresses the "capacity competition" in unified transformers. By assigning independent parameters (QKV and feed-forward layers) to specific modalities, the model can scale more effectively. This allows for asynchronous training and the ability to freeze text-only models while adding new modality-specific parameters.

3. Key Arguments and Research Findings

• The Reasoning Gap: There is a notable asymmetry in transfer learning. While strong language-based reasoning improves image generation (e.g., better detail, less hallucination), training on image/video generation does not necessarily improve a model's general knowledge or reasoning capabilities.
• Language as an Abstraction: Victoria Lin argues that language is a highly compressed abstraction of human cognition, whereas images/videos are passive sensory observations. This fundamental difference explains why next-token prediction on text yields emergent reasoning, while next-frame prediction on video does not yield similar "intelligence" gains.
• Scaling Laws: While text-only models have well-defined power-law scaling, multimodal scaling laws remain under-explored and are a critical area for future research.

4. Notable Quotes

• "Language is fundamentally different from the other modalities because it is a highly compressed abstraction of human cognition." — Victoria Lin, explaining the disparity between text-based reasoning and sensory-based video data.
• "The field found that for image generation and understanding, we sometimes need two types of encodings." — Highlighting the current dilemma in balancing perception and generation.

5. Real-World Applications and Future Directions

• Robotics: The MOT architecture is being adopted in embodied AI to predict action vectors, treating "actions" as a distinct modality.
• Thinking Before Generating: Models like "Bagel" demonstrate that allowing a model to generate "thinking" text before producing an image leads to higher-quality, more detailed outputs.
• Future Research: The field is moving toward unifying image understanding and generation, potentially through more semantic, object-oriented embeddings. There is also a debate on whether "next-token prediction" is the ultimate objective or if more hierarchical, structured learning is required for true physical-world intelligence.

Synthesis/Conclusion

The current state of multimodal AI is characterized by a shift from simple text-only models to "Omni-models" that handle diverse sensory data. While architectures like Transfusion and Mixture of Transformers have significantly improved generation quality and efficiency, a "unified" model that excels equally at perception, reasoning, and generation remains an open challenge. The consensus is that language currently serves as the necessary "skeleton" for reasoning, and future breakthroughs will likely involve better representations of physical-world dynamics and more efficient, modality-aware architectures.