Stanford CS336 Language Modeling from Scratch | Spring 2025 | GPUs

Key Concepts

• GPUs (Graphics Processing Units): Massively parallel processors optimized for throughput, executing the same instruction across many threads.
• SM (Streaming Multiprocessor): An atomic unit within a GPU containing multiple SPs (Streaming Processors) and control logic.
• SPs (Streaming Processors): Execute the same instruction on different data in parallel within an SM.
• Blocks, Warps, and Threads: Hierarchical organization of tasks within a GPU. Blocks are assigned to SMs, warps are groups of 32 threads executing in lockstep.
• Memory Hierarchy: Registers (fastest), local memory, shared memory, global memory (slowest).
• TPUs (Tensor Processing Units): Specialized accelerators similar to GPUs, optimized for matrix multiplication.
• Roofline Model: A performance model showing the limits imposed by memory bandwidth and compute capability.
• SIMT (Single Instruction, Multiple Thread): Execution model where threads in a warp execute the same instruction.
• Lower Precision: Using FP16 or INT8 instead of FP32 to reduce memory bandwidth requirements.
• Operator Fusion: Combining multiple operations into a single kernel to reduce memory transfers.
• Recomputation: Recalculating intermediate values instead of storing them to reduce memory usage.
• Memory Coalescing: Accessing memory in a way that groups accesses from threads in a warp into a single burst read.
• Tiling: Dividing matrices into smaller blocks to improve data locality and reduce global memory accesses.
• Wave Quantization: Performance degradation due to tile sizes not being well-matched to the number of SMs.
• Online Softmax: A method for computing softmax incrementally, tile by tile, without needing the entire input at once.

GPU Architecture and Execution Model

• GPUs are designed for high throughput, prioritizing completing all tasks quickly in aggregate rather than minimizing the latency of individual tasks.
• A GPU consists of many SMs, each containing multiple SPs. An A100 GPU has 128 SMs.
• SMs have control logic and can execute branching instructions, while SPs execute the same instruction on different data.
• Memory hierarchy is crucial: L1 cache and shared memory within SMs are very fast (20 clock cycles), L2 cache is slower, and global memory (DRAM) is much slower (200-300 clock cycles).
• GPU execution model involves blocks, warps, and threads. Blocks are assigned to SMs, warps are groups of 32 threads executing in lockstep.
• Threads within a warp execute the same instruction, which can lead to performance issues with conditional statements.
• Logical memory model includes registers (fastest), local memory, shared memory, and global memory (slowest). Data shared across blocks must be written to global memory.

TPUs

• TPUs are similar to GPUs conceptually, with tensor cores analogous to SMs.
• TPUs have scalar units, vector units, and a specialized matrix multiplication unit (MXU).
• TPUs are optimized for matrix multiplies, making their architecture simpler than GPUs.

GPU Performance Optimization

• The goal is to maximize hardware utilization by avoiding memory bottlenecks.
• The roofline model illustrates the performance limits imposed by memory bandwidth and compute capability.
• Conditionals: Avoid conditional statements within warps, as they can cause threads to be paused.
• Lower Precision: Using FP16 or INT8 instead of FP32 can significantly improve performance by reducing memory bandwidth requirements.
  • Mixed precision training involves using lower precision for inputs and higher precision for intermediate computations.
• Operator Fusion: Combine multiple operations into a single kernel to reduce memory transfers between operations.
  • Tools like torch.compile can automatically fuse operations.
• Recomputation: Recalculate intermediate values instead of storing them to reduce memory usage.
  • Trade compute for memory bandwidth.
  • Gradient checkpointing is a form of recomputation used for memory savings.
• Memory Coalescing: Access memory in a way that groups accesses from threads in a warp into a single burst read.
  • DRAM uses burst mode, where reading one value returns a whole chunk of memory.
  • Accessing memory in column-major order can lead to uncoalesced accesses.
• Tiling: Divide matrices into smaller blocks to improve data locality and reduce global memory accesses.
  • Load tiles into shared memory, perform computations, and then move on to the next tile.
  • Tile sizes must be chosen carefully to avoid underutilization of SMs and to align with burst sections.
  • Padding can be used to ensure that tiles align with burst sections.
• Wave Quantization: Performance degradation due to tile sizes not being well-matched to the number of SMs.

Flash Attention

• Flash Attention accelerates attention by using tiling and recomputation to reduce global memory accesses.
• The key challenge is computing the softmax efficiently.
• Online Softmax: Compute the softmax incrementally, tile by tile, without needing the entire input at once.
  • Maintain running values of the exponentiated sums and maximum terms.
• The forward pass involves tiled matrix multiplies and online softmax computation.
• The backward pass uses recomputation to avoid storing the softmax activations.

Conclusion

Understanding GPU architecture and performance optimization techniques is crucial for maximizing the performance of machine learning workloads. Memory movement is a key bottleneck, and techniques like lower precision, operator fusion, recomputation, memory coalescing, and tiling can be used to reduce global memory accesses and improve performance. Flash Attention is a prime example of how these techniques can be combined to create high-performance algorithms.