Stanford CS336 Language Modeling from Scratch | Spring 2025 | Overview and Tokenization

CS 336: Language Models from Scratch - Lecture 1 Summary

Key Concepts:

• Language Models from Scratch: Building language models end-to-end, including data systems and modeling.
• Leaky Abstractions: The limitations of prompting proprietary models without understanding the underlying technology.
• Industrialization of Language Models: The massive scale and cost of training frontier models like GPT-4.
• Mechanics, Mindset, and Intuitions: Three types of knowledge to be gained in the course.
• Bitter Lesson: Algorithms at scale matter, not just scale itself.
• Efficiency: Maximizing accuracy per resource, crucial for building the best model given a compute and data budget.
• Tokenization: Converting raw text into sequences of integers (tokens).
• BPE (Byte Pair Encoding): A subword tokenization algorithm that learns merges based on corpus statistics.
• Kernels: Optimized code for specific operations on GPUs.
• Parallelism: Distributing model training across multiple GPUs.
• Inference: Generating tokens from a trained model given a prompt.
• Scaling Laws: Relationships between model size, data size, and performance.
• Data Curation: The process of acquiring, cleaning, and preparing data for training.
• Alignment: Making language models useful by fine-tuning them to follow instructions, adopt specific styles, and be safe.
• SFT (Supervised Fine-Tuning): Training a model on prompt-response pairs.
• Learning from Feedback: Improving models using preference data or verifiers.
• DPO (Direct Preference Optimization): A reinforcement learning algorithm for training language models from preference data.
• GRPO (Group Relative Preference Optimization): A simplified and more efficient version of PPO.

Course Overview and Motivation

The course aims to provide a deep, hands-on understanding of language model construction, from data to model deployment. The instructors express concern that researchers are increasingly disconnected from the underlying technology due to reliance on proprietary models. The course philosophy is that to truly understand language models, one must build them from scratch.

• The Problem: Frontier models like GPT-4 (rumored 1.8 trillion parameters, $100 million training cost) are inaccessible for academic research. Small language models may not be representative of larger models due to emergent behaviors and different scaling properties (e.g., the ratio of FLOPs spent in attention vs. MLP layers).
• The Solution: Focus on mechanics (understanding components), mindset (prioritizing hardware efficiency and scaling), and intuitions (data and modeling decisions). While intuitions may not fully translate to larger scales, the course aims to provide a solid foundation.
• The Bitter Lesson Reinterpreted: It's not just about scale, but about algorithms at scale. Algorithmic efficiency improvements have outpaced Moore's Law in image recognition.
• The Goal: Build the best model possible given a specific compute and data budget, emphasizing accuracy per resource.

Historical Context and Current Landscape

Language models have a long history, with early examples including Google's large n-gram models in 2007. The deep learning revolution in the 2010s brought key ingredients like neural language models, sequence-to-sequence models, the Adam optimizer, and the attention mechanism. OpenAI's work on scaling these ingredients led to GPT-2 and GPT-3. The current landscape is dominated by frontier models from companies like OpenAI, Anthropic, and Google, but there's also a growing open-source community.

• Levels of Openness: Closed models (GPT-4), open-weight models (weights available), and open-source models (weights and data available).

Course Logistics and Structure

The course is a five-unit class with a significant workload. It's designed for students with a strong desire to understand the inner workings of language models. The course is not recommended for those primarily interested in the latest techniques or quick results.

• Assignments: Five assignments, each requiring building components from scratch with unit tests and adapter interfaces provided.
• Cluster: Access to a cluster with H100s provided by Together AI.
• AI Tools: Use of AI tools like Copilot is allowed at the student's own risk.
• Grading: Based on unit tests, achieving specific loss levels, and efficiency.

The course is structured into five units:

1. Basics: Implement a basic language model pipeline (tokenizer, model architecture, training loop).
2. Systems: Optimize the pipeline for hardware efficiency (kernels, parallelism, inference).
3. Scaling Laws: Experiment at small scale to predict performance at large scale.
4. Data: Curate and process data for training.
5. Alignment: Fine-tune the model to follow instructions, adopt specific styles, and be safe.

Unit 1: Basics

The goal is to get a basic version of a full pipeline working.

• Tokenizer: Converts strings to sequences of integers. The course focuses on BPE (Byte Pair Encoding).
• Model Architecture: The starting point is the original Transformer. The course will cover improvements like SwiGLU activation, rotary positional embeddings, RMSNorm, and alternatives to the dense MLP.
• Training: Design decisions include optimizer (AdamW), learning rate schedule, and batch size.
• Assignment 1: Implement a BPE tokenizer, Transformer, cross-entropy loss, AdamW optimizer, and training loop. Train on tiny stories and open web text data sets. There will be a leaderboard to minimize open web text perplexity.

Unit 2: Systems

Focuses on optimizing the pipeline for hardware efficiency.

• Kernels: Optimized code for specific operations on GPUs. The course will use Triton to build kernels.
• Parallelism: Distributing model training across multiple GPUs. The course will explore data parallelism and tensor parallelism.
• Inference: Generating tokens from a trained model given a prompt. The course will cover prefill and decode phases, speculative decoding, and system optimizations.
• Assignment 2: Implement a kernel, data parallelism, and a simplified version of model parallelism. Benchmarking and profiling are emphasized.

Unit 3: Scaling Laws

Focuses on experimenting at small scale to predict performance at large scale.

• Chinchilla Optimal: Determining the optimal model size for a given compute budget.
• Scaling Law: Fitting a curve to extrapolate performance at larger scales.
• Assignment 3: Use a provided training API to train models, gather data, fit a scaling law, and predict hyperparameters at a larger scale.

Unit 4: Data

Focuses on curating and processing data for training.

• Evaluation: Evaluating model performance using perplexity, standardized tests (MMLU), and evaluation of instruction following.
• Data Curation: Acquiring, cleaning, and preparing data. This includes crawling the internet, processing HTML and PDFs, filtering, and deduplication.
• Assignment 4: Process raw common crawl data, train classifiers, deduplicate data, and minimize perplexity given a token budget.

Unit 5: Alignment

Focuses on fine-tuning the model to follow instructions, adopt specific styles, and be safe.

• Supervised Fine-Tuning (SFT): Training a model on prompt-response pairs.
• Learning from Feedback: Improving models using preference data or verifiers.
• Algorithms: DPO (Direct Preference Optimization) and GRPO (Group Relative Preference Optimization).
• Assignment 5: Implement supervised tuning, DPO, and GRPO.

Tokenization Deep Dive

Tokenization is the process of converting raw text into sequences of integers (tokens).

• Character-Based Tokenization: Each character is a token. Problem: Large vocabulary size and inefficient use of vocabulary.
• Byte-Based Tokenization: Each byte is a token. Problem: Long sequences and poor compression ratio.
• Word-Based Tokenization: Each word is a token. Problem: Unbounded vocabulary size and handling of unknown words.
• BPE (Byte Pair Encoding): A subword tokenization algorithm that learns merges based on corpus statistics. It starts with byte-level tokens and iteratively merges the most frequent pairs.
• BPE Algorithm:
  1. Convert the string into a sequence of bytes.
  2. Count the occurrences of pairs of adjacent tokens.
  3. Merge the most frequent pair into a new token.
  4. Repeat steps 2 and 3 until the desired vocabulary size is reached.
• Assignment: Implement a fast BPE tokenizer.

Conclusion

The course provides a comprehensive, hands-on approach to building language models from scratch, emphasizing efficiency and a deep understanding of the underlying technology. The goal is to equip students with the skills and knowledge to contribute to fundamental research and build high-performance language models.