Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 15: Mid/Post-Training

Key Concepts

• Post-Training: The process of refining a pre-trained base model to follow instructions and align with human preferences.
• Instruction Following: The ability of a model to interpret and execute complex, multi-step prompts reliably.
• SFT (Supervised Fine-Tuning): Training a model on input-output pairs to mimic desired behaviors.
• RLHF (Reinforcement Learning from Human Feedback): A two-stage process (reward modeling + policy optimization) to align model outputs with human preferences.
• DPO (Direct Preference Optimization): A simplified RLHF algorithm that optimizes the policy directly on preference data without needing a separate reward model or complex PPO training.
• Overoptimization/Reward Hacking: The tendency of models to exploit weaknesses in the reward model or evaluation metrics, leading to "length hacking" or loss of diversity.
• Model Collapse: The loss of output diversity where a model converges to a single, repetitive response style.
• KL Divergence: A regularization term used in RLHF to ensure the fine-tuned model does not deviate too far from the original pre-trained base model.

---

1. The Evolution of Post-Training

The lecture frames post-training as an "artisanal" and "messy" process compared to the systematic nature of pre-training.

• From GPT-3 to Chat-GPT: GPT-3 was a strong base model but lacked reliability for instruction following. Post-training is the bridge that transforms raw "primordial" pre-trained models into useful, steerable assistants.
• The Data Shift: Early approaches like FLAN used existing NLP datasets (e.g., Enron emails, summarization tasks) to create multitask training data. However, these were often unnatural. Modern approaches focus on "chatty" data, synthetic generation, and agentic tool-use data.

2. Supervised Fine-Tuning (SFT)

SFT is primarily a data-engineering challenge.

• Methodology: It involves collecting high-quality input-output pairs. While pre-training requires massive scale, SFT can be effective with fewer, higher-quality examples because the model already possesses latent capabilities from pre-training.
• Data Sources:
  • Self-Instruct: Using models to generate their own high-quality training data.
  • Distillation: Using outputs from stronger models (e.g., GPT-4) to train smaller models (e.g., Alpaca, Vicuna).
  • Crowdsourcing: Projects like Open Assistant attempted to replicate the Wikipedia model of community-driven, high-quality data collection.
• Pitfalls:
  • Hallucination: Training on "tail knowledge" (facts the model doesn't know) combined with specific formatting (e.g., "Reference:") forces the model to hallucinate citations.
  • Style vs. Capability: Models can be "length-hacked" or "style-hacked" to appear smarter by providing bullet points or verbose answers, even if their underlying reasoning hasn't improved.

3. Reinforcement Learning from Human Feedback (RLHF)

RLHF shifts the paradigm from "fitting a distribution" (next-token prediction) to "maximizing a reward."

• The Process:
  1. Sampling: The model generates multiple outputs for a prompt.
  2. Ranking: Human annotators (or reward models) rank these outputs.
  3. Optimization: The model is updated to maximize the reward score.
• Why RL? Humans are better at verifying quality than generating it. RL allows the model to learn from its own outputs and calibrate its confidence, which is harder to achieve via SFT alone.
• Algorithms:
  • PPO (Proximal Policy Optimization): The standard, complex RL algorithm that uses a reward model and importance sampling.
  • DPO (Direct Preference Optimization): A breakthrough that treats RLHF as a classification problem. By assuming a closed-form solution for the optimal policy, DPO optimizes the model by increasing the probability of "winning" responses and decreasing the probability of "losing" ones.

4. The Role of Annotators

• Expertise: There is a shift toward "bespoke" annotation. Companies are hiring domain experts (lawyers, doctors) to ensure high-quality, verifiable data.
• Bias: Annotator demographics significantly influence model behavior. Studies show that models can inherit the ideological biases of their annotators (e.g., shifts in religious or political alignment post-training).
• AI Feedback: Due to cost and scalability, the industry is moving toward using stronger models to annotate data for smaller ones, which has proven as effective as human-only efforts in many benchmarks.

5. Synthesis and Conclusion

The lecture concludes that post-training is a delicate balance between data quality and algorithmic stability.

• Key Takeaway: The boundary between pre-training and post-training is blurring, with many labs now mixing high-quality instruction data into the final "decay" phase of pre-training.
• Future Outlook: While DPO and PPO have made significant strides, the field is moving toward RLVR (Reinforcement Learning with Verifiable Rewards), particularly for reasoning tasks where the model can self-verify its logic, potentially solving the issues of overoptimization and calibration that plague current RLHF methods.