Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 4 - LLM Training

Here's a comprehensive summary of the provided YouTube video transcript:

Key Concepts

• Midterm and Final Exam Logistics: Dates, duration, format, and coverage for the midterm and final exams.
• LLM Training Paradigm: Pre-training and fine-tuning stages.
• Pre-training: Training on vast datasets to predict the next token, focusing on language and code understanding.
• Data for Pre-training: Common Crawl, Wikipedia, social media (Reddit), GitHub, Stack Overflow.
• Compute Metrics: FLOPs (Floating Point Operations) and FLOPS (Floating Point Operations Per Second).
• Scaling Laws: Relationship between model size, data size, and compute for optimal performance.
• Chinchilla Scaling Laws: Optimal ratio of training data size to model parameters (approximately 20:1).
• Pre-training Challenges: Cost, knowledge cutoff date, plagiarism risk.
• Hardware for Training: GPUs and TPUs.
• Training Process: Forward pass, backward pass, weight update, optimizers (Adam).
• Memory Management in Training: Activations, gradients, optimizer states.
• Parallelism Techniques: Data Parallelism (DP), Model Parallelism (Expert, Tensor, Pipeline Parallelism).
• Zero Redundancy Optimization (ZeRO): Sharding optimizer states, gradients, and parameters across GPUs.
• FlashAttention: IO-aware attention mechanism optimizing GPU memory hierarchy (HBM and SRAM) for speed and memory efficiency.
• Recomputation: Discarding activations from the forward pass and recomputing them during the backward pass to save memory.
• Quantization: Reducing the precision of floating-point numbers (e.g., FP32 to FP16) to save memory and increase speed.
• Mixed Precision Training: Using different precisions for weights and activations during training.
• Fine-tuning (Supervised Fine-Tuning - SFT): Adapting a pre-trained model to specific tasks using labeled data.
• Instruction Tuning: A subcategory of SFT to make LLMs helpful assistants by training on instruction-output pairs.
• Data for Instruction Tuning: Story writing, poem creation, list generation, explanations, assistant dialogues, safety-related data.
• Instruction Tuning Data Generation: Using LLMs to generate data, followed by human or LLM review.
• Alignment: The process of making LLMs helpful and harmless, often involving SFT and preference tuning.
• Evaluation of LLMs: Benchmarks (MMLU, GSM8K), human preference (Chatbot Arena), and the challenge of "training on the test task."
• LoRa (Low-Rank Adaptation): Efficient fine-tuning technique by decomposing weight updates into low-rank matrices.
• Quantized LoRa: Applying quantization to LoRa for further memory savings.

---

Logistics and Exam Information

• Midterm Exam:

  • Date: Next week, same time.
  • Duration: 1 hour and 30 minutes (3:30 PM to 5:00 PM).
  • Location: Same classroom.
  • Topics Covered: Lectures 1, 2, 3, and 4.
  • Format: Multiple-choice questions and free-form questions, primarily based on lecture material.
  • Materials Allowed: Pen only; no calculators or notes.
  • Cheat Sheet: Available on the class website for studying but not for the exam.
  • Office Hours: Available after the lecture with Shervin and throughout the week via Ed.
  • Auditors: Inform instructors by the weekend if interested in taking the midterm for printing purposes.

• Final Exam:

  • Date: Wednesday, December 10th.
  • Time: 7:00 PM to 8:30 PM.
  • Location: Different room (not specified in transcript).
  • Topics Covered: Second part of the class, lectures 5 to 9.
  • Format: Closed book, closed notes.

Recap of Previous Lecture (Lecture 3)

• Mixture of Experts (MoE): Architectures (sparse and dense) that activate only a subset of parameters for efficiency, particularly useful for scaling LLMs without high inference costs.
• LLM Next Token Prediction Methods:
  1. Greedy Decoding: Selecting the highest probability token.
  2. Beam Search: Keeping track of the k most probable sequences.
  3. Sampling: Sampling the next token based on the output distribution, with temperature controlling the "spikiness" of the distribution.
• Inference Optimization Techniques: KV Cache mentioned as an important method.

Lecture 4: LLM Training

1. The Paradigm Shift: Transfer Learning and LLM Training Stages

• Traditional ML: Task-specific models trained from scratch for each use case (e.g., spam detection, sentiment analysis).
• Transfer Learning: Leveraging knowledge from a pre-trained model for new tasks, avoiding starting from scratch.
• LLM Training Paradigm:
  1. Pre-training: Training a large model on vast amounts of data to understand language and code.
  2. Fine-tuning (Tuning): Adapting the pre-trained model to specific tasks.

2. Pre-training: The Foundation of LLMs

• Objective: Predict the next token.
• Data: Enormous quantities of text and code from diverse sources.
  • Common Crawl: A massive dataset of web pages (billions per month).
  • Other Sources: Wikipedia, social media (Reddit), code repositories (GitHub), forums (Stack Overflow).
• Data Scale: Hundreds of billions to tens of trillions of tokens.
  • Example: GPT-3 trained on 300 billion tokens; Llama 3 trained on 15 trillion tokens.
• Compute Metrics:
  • FLOPs (Floating Point Operations): A unit of compute. Training LLMs can be on the order of 10^25 FLOPs.
  • Formula: FLOPs are a function of data size (tokens) and model size (parameters), roughly O(tokens * parameters). Architecture (e.g., MoE vs. dense) also influences compute.
  • FLOPS (Floating Point Operations Per Second): A measure of hardware compute speed.
• Scaling Laws (Chinchilla Paper, 2020):
  • Performance improves with more compute, larger datasets, and larger models.
  • Bigger models are more "sample efficient" (better performance per token processed).
  • Optimal Compute Allocation: Given fixed compute, there's a sweet spot for model size and training set size. The "Chinchilla" scaling suggests a training set size about 20 times the model size for optimal compute utilization.
  • Architecture: Transformer-based decoder-only models are standard. Architecture changes are less impactful than model size and data scale.
• Pre-training Challenges:
  • Cost: Millions to hundreds of millions of dollars.
  • Environmental Impact: Significant energy consumption.
  • Knowledge Cutoff Date: Models only know information up to the date their training data was collected. Injecting new knowledge is difficult without degrading performance.
  • Plagiarism: Risk of generating content seen during training.

3. Training LLMs: Hardware and Parallelism

• Hardware: GPUs are essential due to their efficiency in matrix multiplications. Google's TPUs are also used.
• Training Process:
  1. Initialization: Model with billions to hundreds of billions of parameters.
  2. Forward Pass: Input data processed through the network to compute activations. Activations, batch size, and context length (due to O(n^2) self-attention) impact memory usage.
  3. Loss Calculation: Measures the difference between predicted and actual tokens.
  4. Backward Pass: Computes gradients (direction to minimize loss) with respect to each parameter. Gradients also require memory.
  5. Weight Update: Adjusts model weights using an optimizer (e.g., Adam, which uses first and second moments of gradients, also requiring memory).
• Memory Constraints: GPU memory (e.g., 80GB for H100) is limited.
• Parallelism Techniques:
  • Data Parallelism (DP):
    • Distributes data batches across GPUs.
    • Each GPU has a copy of the model.
    • Reduces memory related to batch size.
    • Requires gradient aggregation (communication cost).
    • Limitation: Each GPU must fit the entire model.
  • Zero Redundancy Optimization (ZeRO):
    • Shards optimizer states, gradients, and parameters across GPUs to reduce memory redundancy.
    • ZeRO-1: Shards optimizer states.
    • ZeRO-2: Shards optimizer states and gradients.
    • ZeRO-3: Shards optimizer states, gradients, and parameters.
    • Trade-off: Increased communication costs for reduced memory load.
  • Model Parallelism:
    • Parallelizes operations within a single batch.
    • Expert Parallelism: Distributes MoE experts across GPUs.
    • Tensor Parallelism: Splits large matrix multiplications across GPUs.
    • Pipeline Parallelism: Divides model layers across GPUs, with each GPU processing a stage of the forward pass.

4. FlashAttention: Optimizing Attention Computation

• GPU Memory Hierarchy:
  • HBM (High Bandwidth Memory): Large but slower (tens of GBs).
  • SRAM (Static Random-Access Memory): Much smaller but significantly faster (tens of MBs, 10x faster than HBM).
• Problem with Standard Attention: Frequent reads/writes between HBM and SRAM become a bottleneck, especially due to the row-wise nature of the softmax operation requiring full row computation before normalization.
• FlashAttention Solution:
  • IO-Awareness: Minimizes HBM reads/writes by leveraging SRAM.
  • Tiling: Processes small blocks of matrices (queries, keys, values) in SRAM.
  • Exact Computation: No approximations are made.
  • Key Trick: The softmax of a large matrix can be computed by processing sub-matrices (tiles) and iteratively updating scaling factors, avoiding the need to load the entire matrix into SRAM at once.
  • Benefits: Significantly reduces HBM data transfer, leading to faster computation and memory efficiency.
• Recomputation:
  • During the backward pass, instead of storing all activations from the forward pass (which consumes memory), FlashAttention recomputes them using the fast SRAM.
  • Result: More operations (FLOPs) but significantly fewer HBM reads/writes and reduced runtime.

5. Quantization and Mixed Precision Training

• Quantization: Reducing the precision of floating-point numbers (e.g., from 32-bit to 16-bit) to save memory and increase computation speed.
  • Floating-Point Representation: Bits for exponent, mantissa (granularity), and sign.
  • Common Formats: FP32 (single precision), FP16 (half precision), BF16 (Brain Float 16).
  • Trade-off: Lower precision means less memory and faster computation but potentially reduced accuracy.
• Mixed Precision Training:
  • Concept: Using different precisions for different parts of the model and training process.
  • Typical Strategy: Keep weights in high precision (FP32) and perform forward/backward passes in lower precision (e.g., FP16). Weight updates are then done in FP32.
  • Benefits: Significant memory savings and faster training without substantial performance degradation.
  • Intuition: Weight updates (direction of improvement) may not require extreme precision, while the model's core weights need to maintain higher precision to avoid accumulating quantization errors.
  • Variations: Precision can be applied selectively to different layers or components.
  • Optimal Precision: Not a fixed value; depends on the model and setup. Analysis is often performed on smaller models to determine optimal parameters.
  • Quantization Techniques: Zero-point quantization, Absmax quantization, NF4 (Normal Float 4) for weights.

6. Fine-tuning: Adapting LLMs for Usefulness

• Motivation: Pre-trained models are good at predicting tokens but not necessarily helpful assistants.
• Supervised Fine-Tuning (SFT):
  • Objective: Refine pre-trained weights using labeled data (input-output pairs).
  • Data: High-quality, task-specific datasets.
  • Instruction Tuning: A subcategory of SFT where the model learns to follow instructions.
    • Data Examples: Story writing, poem creation, list generation, explanations, assistant dialogues.
    • Data Generation: Can be human-written or LLM-generated and then reviewed.
    • Safety Data: Includes data to prevent harmful outputs, reject inappropriate prompts, and hedge responses.
  • Process: The model learns to predict the output given an input instruction, with the loss calculated only on the output tokens.
  • Data Scale: Significantly smaller than pre-training data (e.g., 13k examples for GPT-3, 10 million for Llama 3).
• Challenges in SFT:
  • Data Quality and Cost: Requires expensive human annotation and curation.
  • Distribution Mismatch: SFT data distribution might not perfectly match real-world inference prompts, affecting generalization.
  • Memorization: Models might memorize training examples, leading to repetitive or identical responses to similar prompts. Temperature parameter influences this.
  • Evaluation: Subjective and difficult to quantify model performance.
    • Benchmarks: MMLU (Massive Multitask Language Understanding), GSM8K (math reasoning), code generation benchmarks.
    • "Training on the Test Task" Phenomenon: Models can overfit to benchmark-specific data, inflating scores without true capability improvement.
    • Human Preference: Chatbot Arena uses pairwise comparisons to rank models based on user preference, but it's susceptible to manipulation and bias.
    • User Preference vs. Factual Accuracy: Users might prefer helpful but factually incorrect advice.
    • Safety Bias: Users may prefer models that answer prompts over those that adhere to safety principles.
• Alignment: The process of making LLMs helpful and harmless, typically achieved through SFT and preference tuning.
• Mid-training: An emerging technique between pre-training and fine-tuning, aligning pre-training data to specific tasks.

7. Efficient Fine-Tuning: LoRa

• LoRa (Low-Rank Adaptation):
  • Concept: Freezes pre-trained weights (W0) and injects trainable low-rank decomposition matrices (A and B) into specific layers (e.g., attention, feed-forward blocks).
  • Formula: W0 + BA, where BA is the low-rank update.
  • Rank (R): The dimension of the inner matrices A and B, typically much smaller than the original weight dimensions.
  • Benefits: Drastically reduces the number of trainable parameters, leading to significant memory savings and faster fine-tuning.
  • Task Specificity: A and B matrices can be specific to different tasks, allowing for efficient multi-task adaptation from a single base model.
  • Empirical Observations:
    • Higher learning rates (e.g., 10x) are often used.
    • Performance can be negatively impacted by larger batch sizes.
  • Location of LoRa Matrices: Originally applied to attention matrices, but later found to be more beneficial in feed-forward blocks.
• Quantized LoRa:
  • Combines LoRa with quantization.
  • Method: Quantizes the frozen pre-trained weights (W0) using formats like NF4 (Normal Float 4) and trains the LoRa matrices (A and B) in full precision (e.g., BF16).
  • NF4: Assumes normal distribution of weights and uses quantiles for efficient bit allocation.
  • Double Quantization: Further quantizes quantization constants for additional savings.
  • Results: Significant VRAM savings (e.g., 16x).

---

This summary aims to capture the detailed technical information and logical flow of the lecture, maintaining the original language and precision.