Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 6 - LLM Reasoning

Key Concepts

• LLM Reasoning: The ability of Large Language Models to solve problems requiring multi-step thinking processes.
• Chain of Thought (CoT): A prompting technique that encourages LLMs to generate intermediate reasoning steps before providing a final answer.
• Pre-training: The initial, compute-intensive phase of LLM training where models learn language and code structures from vast datasets.
• Fine-tuning (SFT): A subsequent stage where pre-trained models are adapted for specific tasks using high-quality, curated datasets.
• Preference Tuning (RLHF): A method to align LLMs with human preferences, often involving Reinforcement Learning (RL) stages.
• Proximal Policy Optimization (PPO): A common RL algorithm used in LLM alignment, with variants like PPO-Clip and PPO-KLPENALTY.
• Reasoning Models: LLMs specifically designed to output not just answers but also the reasoning process behind them.
• Benchmarks: Standardized tests used to evaluate LLM reasoning capabilities (e.g., HumanEval, GSM8K, AIM).
• Pass@K: A metric to assess the probability of at least one out of K generated attempts succeeding.
• Group Relative Policy Optimization (GRPO): An RL algorithm for LLMs that computes advantages by comparing rewards of multiple completions for the same prompt.
• Distillation: A technique to transfer knowledge from a larger "teacher" model to a smaller "student" model.

Lecture Summary: LLM Reasoning

This lecture delves into the emerging field of LLM reasoning, building upon previous discussions of pre-training, fine-tuning, and preference tuning. The focus is on understanding what reasoning models are, how they are trained, and the challenges associated with their development.

1. Recap of Previous Lectures

• Lecture 4 (Pre-training): The most compute-intensive phase, teaching models language and code structures through large-scale next-token prediction. Results in models capable of autocompletion.
• Lecture 5 (Fine-tuning - SFT & Preference Tuning):
  • SFT (Supervised Fine-Tuning): Adapting pre-trained models for specific tasks (e.g., acting as an assistant) using high-quality, curated datasets.
  • Preference Tuning (RLHF): Aligning models with human preferences, notably using Reinforcement Learning (RL). This involves learning to distinguish good from bad responses and an RL stage to optimize for human preferences, encapsulated in a reward signal.
  • RL Setup Comparison: Traditional RL involves an agent interacting with an environment, receiving rewards. In LLMs, the LLM is the agent, the environment is the set of predictable tokens, and human preferences are the reward.
  • RL Loss Function: Composed of advantage maximization and terms to prevent excessive deviation from previous iterations or the base SFT model.
  • PPO Variants: PPO-Clip and PPO-KLPENALTY were discussed as methods to control policy updates. Modern RLHF often uses a mix of these.

2. Weaknesses of Vanilla LLMs and the Need for Reasoning

Vanilla LLMs, while powerful, exhibit several limitations:

• Limited Reasoning: Struggle with sophisticated problems, especially mathematical or complex logical tasks, due to their next-token prediction objective.
• Static Knowledge: Knowledge is bound by the cutoff date of their pre-training data, making them unaware of recent events.
• "All Talk, No Action": Incapable of performing real-world actions or interacting with external systems.
• Evaluation Challenges: Free-form text generation makes traditional NLP evaluation metrics (like BLEU or ROUGE) insufficient.

The lecture emphasizes that the focus for this session is on improving LLM reasoning capabilities.

3. Defining and Understanding Reasoning Models

• Definition of Reasoning: The ability to solve a problem, typically requiring a multi-step reasoning process. This contrasts with knowledge-based questions.
  • Example (Non-Reasoning): "What is the course code of Stanford's transformers and LM class?" (CME 295)
  • Example (Reasoning): "A bear was born in 2020. How old is it in 2025?" (Requires calculation: 2025 - 2020 = 5 years).
• Core Idea: Chain of Thought (CoT): Leveraging the concept of "thinking in steps" rather than providing a direct answer. CoT was initially introduced through in-context learning examples that explicitly showed reasoning.
• Intuition Behind CoT:
  • Decomposition: LLMs can break down complex problems into more tractable sub-problems, relying on learned patterns.
  • Compute Budget: Generating more tokens (the reasoning chain) effectively provides the LLM with more computational steps to arrive at a solution.
• Reasoning Model Output: Instead of just an answer, reasoning models output both the reasoning chain and the final answer.
• Timeline of Reasoning Models: The field is very recent, with significant developments starting in late 2024 (e.g., OpenAI's GPT-4o preview, Google's Gemini 2.0 Flash, DeepSeek R1 in January 2025).
• Identifying Reasoning Models in Practice: User interfaces often indicate "thinking" time, and some models provide a "thought summary" which is a condensed version of the raw reasoning chain. Users are also charged for reasoning tokens.

4. Benchmarks for Quantifying Reasoning Abilities

• Coding Abilities:
  • Goal: Solve coding problems or fix bugs.
  • Verification: Solutions must pass test cases.
  • Benchmarks: HumanEval, Codeforces, SWBench (derived from GitHub issues).
• Mathematical Abilities:
  • Goal: Solve math problems.
  • Verification: Parsing the generated answer and comparing it to ground truth. Models can be prompted to output answers in a parsable format (e.g., within brackets).
  • Benchmarks: AIM (math exam for US Math Olympiads), GSM8K (grade school math problems).
• Pass@K Metric:
  • Definition: Estimates the probability that at least one of K attempts succeeds.
  • Rationale: Useful when multiple attempts can be afforded, increasing the chance of a correct solution. Analogous to "best of N" sampling.
  • Derivation: The probability of at least one success is 1 minus the probability of all K attempts failing. This involves calculating the probability of sampling K incorrect attempts without replacement from a pool of N observations. The formula derived is: (n-c choose k) / (n choose k), where 'n' is total attempts, 'c' is successful attempts, and 'k' is the number of attempts to consider.
  • Temperature: The temperature parameter in generation influences the diversity of attempts, impacting Pass@K. A moderate temperature (e.g., 0.4-0.8) often balances diversity and performance.
• Other Metrics: Consensus@K, accuracy, exact match.

5. Building Reasoning Models: The Role of Reinforcement Learning

• Challenge: Generating high-quality reasoning chains at scale is difficult. SFT requires extensive human-written reasoning data, which is costly and may not reflect optimal LLM reasoning strategies. The inherent reward signal in reasoning tasks (verifiable correctness) suggests RL is a suitable approach.
• RL Approach:
  • Reward Design:
    1. Formatting Reward: Incentivize the model to produce reasoning chains, often by checking for specific "think" tokens.
    2. Correctness Reward: Verify if the final answer is correct (e.g., passes test cases for code, matches ground truth for math).
  • Performance Improvement: RL with these verifiable rewards has shown significant performance gains on reasoning benchmarks, as demonstrated by the DeepSeek R10 training.
• Controlling Thinking: Research is ongoing to manage the amount of "thinking" a model does to avoid overthinking on simple prompts and to respect context window limitations. Techniques include dynamic budget allocation and "budget forcing."

6. GRPO: Group Relative Policy Optimization

• Motivation: A more recent RL algorithm (2024) designed for reasoning tasks, aiming to simplify the advantage computation compared to PPO.
• Core Idea: Instead of relying on a jointly trained value function (as in PPO), GRPO computes advantages by comparing the reward of a specific completion against the average reward of multiple completions for the same prompt.
• Process:
  1. Generate multiple completions (G) for a given prompt.
  2. Compute rewards for each completion using a reward model or verifiable reward.
  3. Calculate the advantage for each completion as reward - (average_reward / standard_deviation_of_rewards).
  4. Use these advantages to tune the policy model, while also incorporating a KL divergence term to stay close to a reference model.
• Key Difference from PPO: GRPO does not require training a value function. It leverages group-wise reward comparison.
• GRPO Loss Function: Operates on the ratio of current and old policy probabilities, uses clipping for stable updates, and includes a KL divergence term. The advantage computation is the primary differentiator.

7. PPO (Proximal Policy Optimization) for LLMs

• Process:
  1. Generate a single completion for a prompt.
  2. Compute a reward for the entire completion.
  3. Incorporate a KL divergence term (often per token) into the reward calculation.
  4. Train a value function that predicts future rewards.
  5. Use Generalized Advantage Estimation (GAE) to compute the advantage based on rewards and the value function.
  6. Tune the policy model using the computed advantage.
• Models Involved: Policy model (trained), reference model (frozen for KL divergence), reward model (frozen, or verifiable reward), value model (trained).

8. GRPO vs. PPO: A Comparison

| Feature | GRPO | PPO | | :------------------ | :---------------------------------------------------------------- | :------------------------------------------------------------------------------------------------ | | Advantage Calc. | Reward - (Avg Reward / Std Dev Reward) across group completions. | Based on rewards and a jointly trained value function (using GAE). | | Value Function | Not required. | Required and jointly trained. | | Models Trained | Policy model. | Policy model and Value model. |

• Loss Function Similarities: Both operate on policy ratios, use clipping, and include KL divergence (though its placement differs).
• Loss Function Differences: GRPO explicitly includes KL divergence in the objective, while PPO typically incorporates it into the advantage calculation via rewards. The primary difference lies in the advantage computation method.

9. Extensions and Mitigating Output Length Inflation

• Problem: During RL training, LLM output length tends to increase, correlating with performance initially but then plateauing while length continues to grow. This is undesirable for efficiency and cost.
• Hypothesized Cause: The GRPO loss function, by dividing by the output length, can disproportionately downweight tokens in shorter outputs. This incentivizes longer, potentially less optimal, outputs.
• Mitigation Strategies:
  • Equalizing Token Contributions (e.g., DPO): Removing the division by output length or normalizing token contributions across all tokens in a group. This leads to more stable output lengths.
  • Removing the Length Factor: Some approaches completely remove the length normalization factor.
  • Modifications to Standard Deviation: Adjusting the standard deviation calculation in the advantage formula to better account for problem difficulty.
  • Asymmetric Epsilon: Introducing different bounds for policy updates based on token probability to handle low-probability tokens more effectively.

10. DeepSeek R1 and R10: A Case Study in Reasoning Model Training

• R10 (Proof of Concept):
  • Starting Point: A pre-trained model (e.g., DeepSeek V3 architecture with MoE and MLA) trained on next-token prediction.
  • Training: Directly applied RL on reasoning data using verifiable rewards for formatting (e.g., "think" tokens) and correctness.
  • Observation: Achieved significant performance gains on reasoning benchmarks without explicit SFT on reasoning chains.
  • Challenge: Observed issues with language mixing and syntax in reasoning chains, possibly due to lack of strong prior supervision.
• R1 (Full Reasoning Model):
  • Starting Point: DeepSeek V3 base model.
  • Stage 1: Cold Start SFT: Used human-rewritten "coots" (reasoning chains) to address formatting and language consistency issues observed in R10. This stage used a relatively small dataset.
  • Stage 2: RL: Applied RL similar to R10, but with an added "language consistency reward" to further penalize language mixing.
  • Stage 3: Large-Scale SFT: Mixed reasoning data (generated via rejection sampling with LLM judges) and non-reasoning data (recycled from V3) in a 3:1 ratio. This stage aimed to improve overall usefulness.
  • Stage 4: Final Alignment: Mixed reasoning and non-reasoning data, aligning for helpfulness and harmlessness (reward applied to all tokens, including thought chains).
  • Results: R1 achieved competitive performance with closed-source reasoning models and demonstrated the effectiveness of the multi-stage pipeline.

11. Distillation for Smaller Reasoning Models

• Concept: Applying distillation techniques to transfer knowledge from a large teacher model (like R1) to a smaller student model.
• Method:
  1. Offline Generation: Use the teacher model (R1) to generate sample responses, including reasoning chains.
  2. Student Training: Train a smaller student model to predict the entire sequence of tokens generated by the teacher, rather than just the next token probability distribution.
• Benefits: This approach is more efficient for smaller models than training from scratch using RL techniques, achieving competitive results with models like "01 mini."

Conclusion

LLM reasoning is a rapidly evolving area. The lecture highlights the shift from vanilla LLMs to models capable of explicit reasoning, driven by advancements in RL techniques like GRPO and sophisticated multi-stage training pipelines. The development of benchmarks and metrics like Pass@K is crucial for evaluating progress. The DeepSeek R1 and R10 case studies illustrate how RL, combined with careful data curation and multi-stage alignment, can lead to powerful reasoning models, and distillation offers a path to democratize these capabilities for smaller models.