Stanford CME296 Diffusion & Large Vision Models | Spring 2026 | Lecture 8 - Trending Topics

Key Concepts

• Diffusion Models: Generative models that learn to reverse a noise-adding process to transform Gaussian noise into data.
• Score-Based Generative Modeling: Estimating the gradient of the log-data density (the "score") to guide samples toward the data distribution.
• Flow Matching: A mass-transport framework that learns a vector field to map an initial distribution to a target distribution via Ordinary Differential Equations (ODEs).
• Latent Space: A compressed, lower-dimensional representation of data (often learned via VAEs) that simplifies generation.
• Diffusion Transformers (DiT): Architectures replacing U-Nets with self-attention mechanisms to handle global dependencies in image generation.
• Distillation: Techniques to reduce the number of inference steps required for generation.
• Model Collapse: A phenomenon where models trained on AI-generated data degrade in quality over time.

---

1. Recap of Image Generation Paradigms

The course structured image generation into three primary mathematical frameworks:

• Diffusion (DDPM): Uses a forward process to add Gaussian noise and learns to predict the noise to remove. It relies on the Evidence Lower Bound (ELBO) for tractable training.
• Score-Based Models: Uses Langevin Dynamics to sample from the data distribution by following the "score" (gradient of log-density). It introduces a noise-conditional score matching loss to handle the trade-off between noise levels and estimation accuracy.
• Flow Matching: Frames generation as mass transport. It learns a velocity field ($u_t(x)$) to move probability mass from an initial distribution to a target distribution. This is currently the industry standard due to its efficiency.

2. Representation and Architecture

• Latent Space & VAEs: To handle high-dimensional pixel data, models use Variational Autoencoders (VAEs) to compress images into a compact latent space. This makes the generation task more "tractable" by removing redundant information.
• Architectural Evolution: The field has shifted from U-Nets (downsampling/upsampling with skip connections) to Diffusion Transformers (DiT). DiTs use self-attention to allow long-range interactions between image patches, which is critical for complex compositions.
• Training Strategies: Modern training uses non-uniform time-step sampling (e.g., logit-normal distribution) to focus on "hard" noise levels. High-resolution training requires increased noise levels to account for spatial correlation.

3. Evaluation Metrics

• ELO Rating: Used for pairwise model comparisons. It accounts for the strength of the opponent, preventing bias when a model wins against weak competitors.
• FID (Fréchet Inception Distance): Measures the distance between the distribution of generated images and real images. It assumes Gaussian distributions for the feature representations.
• LLM-as-a-Judge: Using Multi-modal Large Language Models (MLLMs) to evaluate image quality and alignment with prompts, creating a faster feedback loop than human evaluation.

4. State-of-the-Art Trends

• Scaling vs. Compression: Recent models (e.g., top-ranked models from 2026) are experimenting with pixel-space generation using large patch sizes (32x32) and massive parameter scaling (up to 200B parameters), potentially bypassing the need for VAEs.
• Text Encoders: While many models use pre-trained text encoders (like CLIP or Mistral), some newer approaches train custom text encoders alongside the generation model to better align with specific task requirements.

5. Adjacent Fields and Future Applications

• Video Generation: Extends image generation by adding a temporal dimension. It utilizes 3D Causal VAEs to compress both space and time. Temporal consistency is maintained by ensuring the receptive field of the model depends on past frames (causality).
• Image Editing: Rather than generating from scratch, modern approaches use VLM-guided editing actions (e.g., "decrease brightness by 50%") to preserve the original image structure.
• Diffusion for Text: A shift from auto-regressive (sequential) generation to "filling in the blanks" via diffusion. This allows for parallel generation and is particularly effective for "fill-in-the-middle" coding tasks, offering potential 10x speedups.

6. Challenges and Synthesis

• Data Quality & Trust: The risk of "model collapse" (training on AI-generated data) is a major concern. Solutions include C2PA metadata and watermarking (e.g., SynthID) to track provenance.
• Hardware: Current hardware is optimized for matrix multiplication, but future research is exploring analog-to-numeric simplifications for attention mechanisms.
• Conclusion: The field is moving toward multi-modal synthesis. While image generation is largely "solved" in terms of quality, the next frontiers involve reasoning, robotics, and efficient, constraint-based editing. The instructors emphasize that understanding the underlying math (Flow Matching) and cloning GitHub repositories for hands-on experimentation are the best ways to stay current in this fast-moving field.