Stanford CS336: Language Modeling from Scratch | Spring 2025 | Pytorch, Resource Accounting

Key Concepts

• Tensors: Multidimensional arrays, the fundamental data structure in PyTorch.
• Floating-point precision (FP32, FP16, BF16, FP8): Different ways to represent numbers, affecting memory usage, speed, and dynamic range.
• Flops (Floating-point operations): A measure of computational work.
• MFU (Model Flops Utilization): A metric for how efficiently hardware is used.
• Inops: Operations using named dimensions instead of indices for better code readability and maintainability.
• Gradients: The derivatives of the loss function with respect to the model's parameters, used for updating the parameters during training.
• Optimizers: Algorithms that update the model's parameters based on the gradients.
• Activation Checkpointing: A technique to reduce memory usage by recomputing activations during the backward pass instead of storing them.
• Mixed Precision Training: Using different floating-point precisions for different parts of the model to balance speed and accuracy.

Memory Accounting

• Tensors as Building Blocks: Parameters, gradients, optimizer states, data, and activations are all stored as tensors.
• Floating-Point Representation:
  • Float32 (FP32): 32 bits, the default, offers good precision and dynamic range.
  • Float16 (FP16): 16 bits, halves memory usage but has limited dynamic range, leading to underflow/overflow issues.
  • BFloat16 (BF16): 16 bits, prioritizes dynamic range over precision, making it suitable for computations.
  • FP8: 8 bits, offers the smallest memory footprint but can introduce instability.
• Memory Calculation: Memory usage is determined by the number of values in the tensor and the data type of each value.
  • Example: A 4x8 float32 tensor uses 4 * 8 * 4 bytes = 128 bytes.
• Views and Contiguity:
  • Views are different ways of looking at the same underlying data without copying.
  • Operations like slicing, view(), and transpose() can create views.
  • contiguous() creates a copy if the tensor is not contiguous in memory.

Compute Accounting

• GPU Acceleration: Moving tensors to the GPU is crucial for performance.
• Tensor Operations: Operations on tensors have memory and compute footprints.
• Tensors as Pointers: Tensors in PyTorch are pointers to allocated memory, with metadata (strides) defining how to access elements.
• Inops for Clarity:
  • Inspired by Einstein summation notation.
  • Uses named dimensions instead of indices, improving code readability.
  • Libraries like Jack typing can be used to specify dimension types.
  • einsum performs matrix multiplication with good bookkeeping.
  • reduce aggregates dimensions of a tensor.
  • rearrange reshapes tensors.
• Flops as a Metric:
  • Floating-point operations (addition, multiplication) are the main contributors to flop count.
  • Distinction between "flops" (number of floating-point operations) and "flops/s" (floating-point operations per second).
• Linear Model Example:
  • A linear model maps a d-dimensional vector to a k-dimensional vector.
  • The number of flops for the forward pass is approximately 2 * B * D * K (where B is the batch size).
• Model Flops Utilization (MFU):
  • MFU = (Actual Flops / Time) / Promise Flops per Second.
  • A measure of how efficiently the hardware is being used.
  • MFU > 0.5 is generally considered good.

Gradients and Backward Pass

• Gradient Computation: Computing gradients adds to the overall computational cost.
• Two-Layer Linear Model Example:
  • Forward pass: X -> H1 -> H2 -> Loss
  • Backward pass: Compute gradients with respect to H1, H2, W1, and W2.
  • The number of flops for the backward pass is approximately four times the number of parameters.
  • Total flops (forward + backward) is approximately six times the number of parameters.

Building Models

• nn.Parameter: PyTorch objects for storing model parameters.
• Parameter Initialization:
  • Random initialization can lead to unstable training.
  • Xavier initialization (scaling by 1/sqrt(number of inputs)) helps stabilize training.
• Custom Model Creation: Creating custom models using nn.Module and defining the forward pass.
• Randomness and Determinism:
  • Fixing random seeds is crucial for reproducibility.
  • Use different random seeds for different sources of randomness.
• Data Loading:
  • memmap allows loading large datasets on demand.
  • Create a data loader to sample data in batches.

Optimizers

• Optimizer Algorithms: Stochastic Gradient Descent, Momentum, Adagrad, RMSprop, Adam.
• Custom Optimizer Implementation:
  • Override the Optimizer class.
  • Implement the step() method to update parameters.
  • Store optimizer state in a dictionary.
• Memory Requirements of Optimizer States:
  • Adagrad requires storing the sum of gradient squares, adding to memory usage.
• Total Memory Calculation:
  • Total memory = Memory for parameters + Memory for activations + Memory for gradients + Memory for optimizer states.

Training Loop

• Typical Training Loop:
  • Define the model and optimizer.
  • Load data.
  • Perform forward pass, backward pass, and optimizer step.
• Checkpointing:
  • Periodically save the model, optimizer, and iteration number to disk.
• Mixed Precision Training:
  • Use lower precision (BF16, FP8) for the forward pass and higher precision (FP32) for the rest.
  • PyTorch provides tools for automatic mixed precision training.

Conclusion

The lecture provides a comprehensive overview of the fundamental concepts and techniques for building and training deep learning models in PyTorch. It emphasizes the importance of resource accounting (memory and compute) and provides practical guidance on optimizing model training for efficiency. The discussion covers tensors, floating-point precision, flops, MFU, inops, gradients, optimizers, and mixed precision training. The lecture also highlights the interplay between hardware capabilities and model design, suggesting that future model architectures may be heavily influenced by the characteristics of available hardware.