Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 15: Alignment - SFT/RLHF

Key Concepts

• Post-training: Techniques to make pre-trained language models useful and safe.
• RLHF (Reinforcement Learning from Human Feedback): A method to align language models with human preferences using reinforcement learning.
• Supervised Fine-tuning (SFT): Training a pre-trained model on expert demonstrations.
• Instruction Tuning: Training a language model to follow instructions.
• Pairwise Feedback: Comparing two model outputs and selecting the better one.
• Reward Model: A model that predicts a scalar reward for a given model output.
• Policy Optimization (PO): A reinforcement learning algorithm used in RLHF.
• Direct Preference Optimization (DPO): A reinforcement learning algorithm that simplifies RLHF by directly optimizing for preferences.
• Mid-training: Mixing instruction tuning data into the tail end of pre-training.
• Hallucination: A language model generating incorrect or nonsensical information.
• Safety Tuning: Training a language model to avoid generating harmful or inappropriate content.
• Off-policy RL: Reinforcement learning where the data is collected from a different policy than the one being trained.
• On-policy RL: Reinforcement learning where the data is collected from the same policy that is being trained.

1. Introduction to Post-Training and RLHF

• The lecture focuses on post-training techniques, specifically RLHF, to transform pre-trained models like GPT-3 into useful and safe systems like ChatGPT.
• The goal is to enable tighter control over language models by collecting data of desired behaviors and training the model to exhibit those behaviors.
• Key questions addressed: What does the data look like? How hard is it to collect? How do we use the data algorithmically? How do we scale this up?

2. Supervised Fine-Tuning (SFT)

• SFT involves training a pre-trained model on expert demonstrations.
• Two key ingredients for successful SFT:
  • Training data: High-quality expert demonstrations.
  • Method: Gradient descent, but with considerations for scaling and potential surprises.
• Different paradigms for constructing instruction tuning data:
  • Flan: Aggregating existing NLP datasets.
    • Pros: Large amounts of data can be obtained relatively easily.
    • Cons: Data can be unnatural and require significant surgery to fit the instruction tuning format.
  • Alpaca: Using language models to generate instruction tuning data.
    • Pros: Data looks more like standard chatbot inputs.
    • Cons: Inputs may lack diversity and be too short.
  • Open Assistant: Human-written instruction tuning data.
    • Pros: High-quality, detailed responses.
    • Cons: Difficult and costly to collect.

3. Interactive Exercise: Instruction Tuning Data Collection

• An interactive exercise was conducted where participants were asked to provide an instruction tuning response to a given prompt.
• The exercise highlighted the difficulty of writing long-form responses and the challenges of crowd-sourcing high-quality data.
• AI feedback is becoming popular due to its ability to generate long, detailed responses at a lower cost than human annotation.

4. Considerations for Instruction Tuning Data

• Important factors to consider when collecting instruction tuning data:
  • Length: Length biases can affect human and AI evaluations.
  • Knowledge: Including deep knowledge and citations can lead to hallucination if the model doesn't already possess that knowledge.
  • Safety: Safety tuning is crucial to prevent models from generating harmful or inappropriate content.
• A counterintuitive phenomenon: Fully correct and rich instruction tuning data can be detrimental if it encourages the model to make up facts.
• Safety tuning involves a trade-off between refusing unsafe responses and avoiding excessive refusal.

5. Scaling Up Instruction Tuning

• Modern instruction tuning pipelines are increasingly resembling pre-training pipelines.
• A popular approach is to mix instruction tuning data into the tail end of pre-training (mid-training).
• This allows for scaling up without catastrophic forgetting and can improve data leverage.
• Example: MiniCPM uses a two-stage training pipeline with pure pre-training followed by a decay stage with mixed pre-training and instruction tuning data.
• The blurring of boundaries between pre-training and instruction tuning makes it difficult to reason about the true nature of "base models."

6. Reinforcement Learning from Human Feedback (RLHF)

• RLHF shifts the perspective from generative modeling to policy optimization, where the goal is to find a policy that maximizes rewards.
• Two main reasons for using RLHF:
  • SFT data is expensive to collect.
  • Verifying outputs is often easier and potentially higher quality than generating them.
• RLHF process:
  1. Model generates outputs (rollouts).
  2. Outputs are compared pairwise.
  3. A reward model is trained to predict scalar rewards for each output.
  4. Reinforcement learning is used to train the model to maximize the reward model's predictions.

7. Pairwise Feedback Data Collection

• Pairwise feedback involves comparing two model outputs and selecting the better one.
• Annotation guidelines for pairwise feedback typically focus on helpfulness, truthfulness, and harmlessness.
• Interactive exercise: Participants were asked to judge which of two responses was better, highlighting the difficulty of fact-checking and the potential for disagreement.
• Challenges in collecting pairwise feedback:
  • Difficulty of getting high-quality, verifiable annotators.
  • Time constraints for annotators.
  • Potential for annotators to use language models to generate responses.
  • Ethical concerns related to outsourcing annotation to third countries.
• RLHF and alignment have a strong influence on model behaviors due to their position at the end of the pipeline.
• Annotator biases can affect the alignment of the model.
• AI feedback is increasingly used in RLHF due to its cost-effectiveness and comparable agreement with human feedback.
• Length effects can confound preference judgments.

8. RLHF Methods: PO and DPO

• The goal of RLHF is to find a policy that maximizes rewards.
• The InstructGPT paper defines an objective that includes a reward term and a KL divergence term to prevent the RL policy from deviating too far from the SFT model.
• Policy Optimization (PO):
  • Involves calculating an advantage (a variance-reduced version of the reward).
  • Uses importance weighting corrections to account for multiple gradient steps.
  • Clips probability ratios to incentivize the model to stay close to the original policy.
• Direct Preference Optimization (DPO):
  • Simplifies RLHF by directly optimizing for preferences.
  • Removes the reward model and on-policy components of PO.
  • Takes gradient steps on the log loss of good outputs and negative gradient steps on the log loss of bad outputs.
• DPO derivation:
  1. Assume the policy is an arbitrary function.
  2. Parameterize the reward via the policy.
  3. Optimize the policy using supervised losses based on pairwise comparisons.

9. Conclusion

• The lecture provided a detailed overview of post-training techniques, focusing on SFT and RLHF.
• It highlighted the importance of data quality, the challenges of data collection, and the algorithmic considerations for training language models to align with human preferences.
• The lecture also discussed the trade-offs involved in safety tuning and the potential for biases to affect model behavior.
• Finally, the lecture introduced two RLHF algorithms, PO and DPO, and derived the DPO formula.