Stanford CS336 Language Modeling from Scratch | Spring 2026 | Guest Lecture: Dan Fu

Key Concepts

• Inference Engine: The system responsible for converting electricity (compute) into intelligence (tokens).
• Prefill vs. Decode: The two distinct phases of inference; prefill is compute-bound (processing input tokens), while decode is memory-bandwidth-bound (generating one token at a time).
• KV Cache: A data structure storing previous token activations to avoid redundant computation; critical for performance in multi-turn conversations.
• Continuous Batching: A scheduling technique that processes multiple requests simultaneously to maximize GPU utilization.
• Mega Kernels: A kernel optimization strategy that fuses multiple operations into a single GPU kernel to reduce launch overhead and improve hardware utilization.
• ThunderKittens: A low-level CUDA framework for fine-grained GPU control and kernel development.
• Parse (Recurrent Transformers): A model architecture that uses looped blocks to increase expressivity and quality without increasing parameter counts.
• Spectral Radius: A mathematical metric used to analyze and stabilize the training of recurrent/looped models by constraining activation growth.

---

1. The Inference Lifecycle and System Architecture

The speaker emphasizes that inference is the "engine" of modern AI. When a request enters the system:

• Scheduling: Requests are routed to GPUs, often separating prefill and decode workloads onto different hardware to optimize for their specific bottlenecks (compute-heavy vs. memory-bandwidth-heavy).
• KV Cache Management: Systems use prefix sharing to cache activations. As GPU memory fills, systems offload to CPU DRAM and eventually SSDs, mirroring classic Operating System memory management (paging/swapping).
• Continuous Batching: To handle production traffic, engines interleave requests, managing memory and compute resources dynamically as new tokens are generated.

2. Research and Optimization: Mega Kernels

The speaker highlights the inefficiency of standard kernel execution, where "kernel launch" and "tail effects" create significant downtime.

• Methodology: Instead of writing kernels for individual operations (e.g., norm, attention), "Mega Kernels" fuse entire layers or multiple operations into one.
• Evidence: This approach allows for overlapping weight loads and computation, achieving up to 72% bandwidth utilization on H100 GPUs—near the "speed of light" for the hardware.
• Trade-offs: Mega kernels are extremely labor-intensive to write and maintain, requiring specialized engineering expertise for every model and hardware configuration.

3. Architectural Innovation: Parse (Recurrent Transformers)

The speaker introduces Parse, a model architecture that uses recurrent loops to improve quality per parameter.

• The Problem: Naive loop transformers suffer from "loss spikes" and instability during training.
• The Solution: By modeling the residual as a dynamic system and constraining the spectral radius of the transformation matrices ($A$ and $B$) to be less than one, the team successfully stabilized training.
• Scaling Laws: Research suggests that as data increases, one should scale both parameters and the number of recurrences. This implies that current non-recurrent models may be missing out on potential quality gains.

4. Real-World Applications and Challenges

• Workload Variability: Coding agents (e.g., Cursor) involve long input contexts and iterative tool-calling, whereas standard chat is turn-based. These require different serving strategies.
• Production Bugs: Large-scale systems encounter rare, subtle bugs, such as "off-by-one" errors in kernels leading to unexpected language shifts (e.g., outputting Chinese characters) or infinite "doom loops" in tool-calling.
• Hardware Co-design: Future architectures are being designed with specific hardware in mind, such as utilizing proprietary formats like Nvidia’s NV FP4 or specialized chips like Groq’s LPU for decode-heavy tasks.

5. Notable Quotes

• "Inference is the engine that turns electricity into intelligence."
• "If you understand inference and understand the inference engines... you can enable full stack innovation in machine learning algorithms."
• "[Mega kernels] are very, very labor intensive to write... you will never be able to go faster, but it just takes a lot of energy and a lot of effort."

Synthesis

The core takeaway is that the "industrial revolution" of AI is shifting from simple training to complex inference optimization. By treating the GPU as a programmable distributed system rather than a black box, researchers can achieve massive performance gains through Mega Kernels and architectural innovations like Parse. The future of AI efficiency lies in the deep co-design of hardware, kernels, and model architectures, moving beyond the current standard of non-recurrent, non-fused transformer models.