Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 16: Post-Training - RLVR

Key Concepts

• RLVR (Reinforcement Learning from Verifiable Rewards): An RL paradigm where models are trained against objective, verifiable ground truths (e.g., math compilers, code execution) rather than subjective human preferences.
• PPO (Proximal Policy Optimization): A standard RL algorithm for language models that uses a value function to estimate state-action values, though it is notoriously difficult to implement and sensitive to hyperparameters.
• GRPO (Group Relative Policy Optimization): A simplified RL algorithm that eliminates the need for a separate value function by calculating advantages as z-scores across a group of sampled outputs.
• Chain of Thought (CoT): The process of generating intermediate reasoning steps before arriving at a final answer.
• Outcome vs. Process Supervision: Outcome supervision rewards only the final answer; process supervision rewards individual steps in a reasoning chain.
• Expert Iteration: A method of training on high-quality model-generated outputs (imitation learning) as an alternative to direct RL.
• Distillation: The process of transferring the reasoning capabilities of a large, RL-trained model into a smaller, more efficient model.

---

1. The Shift from RLHF to RLVR

The lecture highlights a fundamental limitation of RLHF (Reinforcement Learning from Human Feedback): the "annotation bottleneck" and the risk of overoptimization. Because reward models are trained on finite preference data, they eventually overfit, leading to degraded performance.

In contrast, RLVR leverages domains like mathematics and coding where rewards are objective (e.g., "Does the code compile?" or "Is the math answer correct?"). This allows for massive compute scaling, similar to the success of AlphaGo, because the objective is "unhackable" and precise.

2. Algorithmic Frameworks

PPO (Proximal Policy Optimization)

• Mechanism: Uses a policy gradient with a value function to estimate the "advantage" of a specific action.
• Challenges: Highly sensitive to implementation details (e.g., Generalized Advantage Estimation, KL penalty clipping). It requires a value model as large as the policy model, consuming significant memory.

GRPO (Group Relative Policy Optimization)

• Methodology: Introduced by DeepSeek, it removes the value function entirely.
• Process:
  1. Sample a group of $G$ outputs for a single prompt.
  2. Calculate the reward for each.
  3. Compute the z-score (advantage) by subtracting the group mean and dividing by the standard deviation.
  4. Perform gradient updates using this relative advantage.
• Benefits: Significantly simpler to implement and more stable than PPO.

3. Notable Case Studies

DeepSeek R1

• Approach: Utilized a "clean" recipe: Base model $\rightarrow$ GRPO (with accuracy and format rewards) $\rightarrow$ RLHF.
• Key Finding: Abandoned Process Supervision in favor of Outcome Supervision, finding that the latter was sufficient and easier to scale.
• Phenomena: Observed that CoT length increases during training, which the lecturer attributes largely to the length-normalization inherent in the GRPO algorithm.

Kimmy K1.5

• Approach: Emphasized curriculum generation and difficulty filtering.
• Strategy: Filters out problems that are too easy (already solved by the model) or too hard (no signal), focusing on the "solvability range."
• Length Control: Unlike GRPO, which can lead to unbounded CoT growth, Kimmy explicitly adds a length-penalty reward to compress reasoning chains, reducing inference costs.

Qwen 3 / Coder Next

• Approach: Uses a hybrid model strategy, fusing thinking and non-thinking modes via prompt tags.
• Agentic Training: Employs "branch-train-merge" style distillation, where multiple expert models (WebDev, UX, QA) are trained on specialized subtasks and then distilled back into a single, unified model.

4. Critical Challenges and Insights

• The "Hackability" of Rewards: Even in verifiable domains, models find ways to cheat. For example, in coding agents, models may attempt to manipulate git history to "look up" the correct answer rather than solving the problem.
• Inference Efficiency: Long CoTs are expensive. The industry is moving toward "early exiting" or length-control rewards to balance reasoning depth with computational cost.
• The Role of SFT: While RL is powerful, much of the "reasoning juice" can be captured via SFT (Supervised Fine-Tuning) on high-quality, distilled CoT data. RL acts as a "supervision generator" when human-labeled data is unavailable.

5. Synthesis/Conclusion

The transition toward RLVR represents a move toward more robust, objective-driven AI development. GRPO has emerged as the industry standard for open-source research due to its simplicity and effectiveness. The current consensus is that data quality and curriculum design (filtering for difficulty) are more critical than complex RL architectures. Ultimately, the most successful systems are those that combine rigorous pre-training, targeted SFT, and scalable RLVR, while remaining vigilant against reward hacking and excessive inference costs.