Stanford CS336 Language Modeling from Scratch | Spring 2025 | Scaling laws

Scaling Laws Summary

Key Concepts: Scaling laws, data scaling, model scaling, compute scaling, power law, irreducible error, excess error, isoflop analysis, Chinchilla ratio, critical batch size, MUP (Maximal Update Parameterization), downstream tasks, joint scaling laws.

1. Introduction to Scaling Laws

• Scenario: Given abundant compute (e.g., 100,000 H100s for a month), how to build the best open-source LLM.
• Problem: Moving beyond copying existing models (e.g., LLaMA) to innovate and optimize solutions.
• Scaling Laws Defined: Simple, predictive laws for language model behavior, enabling extrapolation from small models to larger ones.
• Old vs. New Optimism:
  • Old: Train large models, tune hyperparameters (compute-intensive).
  • New: Train small models, learn from them, extrapolate to larger models (more efficient).
• Contextualization: Scaling laws are not just about log-linear relationships leading to AGI but are grounded in empirical observations and have a rich history.

2. History and Background of Scaling Laws

• Statistical Machine Learning Connection: Scaling laws relate data/model size to expected model behavior, similar to VC dimension and Rademacher complexity in theory.
  • Example: Generalization bound scales as 1/sqrt(m).
  • Example: L2 error of density estimation bounded by n^(beta/(2*beta+1)).
• Leap from Theory to Empirical: Scaling laws bridge the gap between theoretical bounds and empirical observations.
• Early Scaling Law Paper (1993, Bell Labs):
  • Proposed a method to predict model performance without full training.
  • Functional form: Test error = irreducible error + polynomially decaying term.
  • Trained small models, fit curves, and predicted behavior further out.
• Bo Banko and Bril: Studied how NLP system performance scales with data.
  • Argued for dramatic performance improvements through data scaling.
  • Considered the trade-off between algorithm development and data collection.
• Predictability and Functional Forms: Early discussions on whether power laws are the right functional forms.
• Hess et al. (2017): Showed error rates fall as a power law for machine translation, speech, and vision tasks.
  • Identified three regions: best guess, predictable scaling (power law), and asymptotic (irreducible error).
  • Recognized the importance of computational limits and potential for quantization.

3. Data Scaling Laws

• Definition: A formula mapping dataset size (n) to excess error.
• Empirical Observation: On a log plot, model performance is linear with dataset size (power law or scale-free).
• Implication: Polynomial relationship between dataset size and error.
• Example: Estimating the Mean:
  • Input from a Gaussian distribution.
  • Error scales as sigma^2/n (polynomial in n).
  • Log-log plot yields a slope of -1.
• Nonparametric Function Classes: Expect dimension dependence, leading to slower scaling.
  • Example: Estimating a regression function in a 2D unit box.
  • Error scales as n^(-1/d), where d is the dimensionality.
• Intrinsic Dimensionality: The slope of the scaling law reflects the intrinsic dimensionality or ease of learning the task.
• Engineering Decisions with Data Scaling Laws:
  • Dataset Composition: Affects the offset, not the slope, allowing data selection experiments on smaller models.
  • Multi-Epoch Training: Diminishing returns after about four epochs, leading to the concept of effective sample size.
  • Data Selection in Large Data Regime: Balancing repeating high-quality sources vs. including new data.

4. Model Scaling Laws

• Practical Engineering Questions: Which architectures, optimizers, and hyperparameters are worth scaling up?
• Kaplan Scaling Law Paper: A comprehensive study of various scaling observations.
• Architecture Comparison (Transformers vs. LSTMs):
  • Train LSTMs and transformers across compute levels.
  • Transformers show a constant factor compute advantage over LSTMs.
• Optimizer Choice (SGD vs. Adam): Adam shows a constant factor compute advantage over SGD.
• Depth vs. Width: Scaling law analysis backs up the idea that there is a wide basin of near-optimal aspect ratios.
• Subtlety: Not All Parameters Are Equal: Embedding parameters don't scale the same way as non-embedding parameters.
• Hyperparameter Tuning: Be scale-aware when tuning hyperparameters.
• Batch Size:
  • Diminishing returns past a certain point.
  • Critical batch size: The threshold where scaling becomes ineffective.
  • Critical batch size tends to get smaller as loss target gets better.
• Learning Rate:
  • Optimal learning rate varies with model scale.
  • Rule of thumb: Learning rate scales as 1/width.
  • MUP (Maximal Update Parameterization): Aims to reparameterize the model for stable learning rates across scales.

5. Joint Data Model Scaling Laws

• Core Question: Do we need more data or bigger models?
• Joint Scaling Laws: Equations describing the trade-off between data and model size.
  • Example (Rosenfeld): Error = term decaying with data + term decaying with model size + irreducible error.
• Chinchilla Paper: Empirically determined the optimal trade-off between tokens and model size for minimal training flops.
  • Chinchilla ratio: ~20 tokens per parameter.
• Learning Rate Schedules: Cosine learning rates must be trained to the end for valid models.
• Chinchilla Methods:
  • Method 1 (Minimum over Curves): Identify the lower envelope of training curves and fit a scaling law.
  • Method 2 (Isoflop Analysis): Pick compute scales, sweep over model sizes, and find the minimum loss for each scale.
  • Method 3 (Curve Fitting): Fit the joint scaling functional form to the data.
• Epoch AI Replication of Method 3: Corrected a curve-fitting error, leading to results consistent with methods 1 and 2.
• Shift to Inference Costs: Focus on intelligence at smaller parameter sizes, leading to higher tokens per parameter ratios.

6. Scaling Laws in Other Domains

• Diffusion Models: Isoflop analysis yields similar curves to auto-regressive models, demonstrating the broad applicability of scaling laws.

7. Cautions and Considerations

• Downstream Tasks: Scaling laws are well-behaved for log losses but less predictable for downstream tasks.
• Perplexity Scaling vs. Downstream Scaling: Don't assume perplexity scaling translates directly to downstream performance.

8. Scaling Law-Based Design Procedure

1. Train a few smaller models spanning several orders of magnitude compute.
2. Establish a scaling law (log-log linear relationship).
3. Set optimal hyperparameters based on the prediction.

9. Conclusion

Scaling laws provide a framework for optimizing various aspects of model design and resource allocation. They enable extrapolation from small-scale experiments to large-scale models, facilitating informed decisions about hyperparameters, architectures, and data usage. The Chinchilla analysis highlights the importance of balancing model size and data quantity for optimal performance within a given compute budget.