Pushing the capabilities of Gemma 3 via distillation and RL fine-tuning

Gemma 3: Distillation and RL Fine-Tuning for Enhanced Capabilities

Key Concepts:

• Distillation: Transferring knowledge from a larger, more capable model (teacher) to a smaller model (student).
• RL Fine-Tuning: Reinforcement Learning used to further refine the model's behavior based on specific reward functions.
• Symbolic Matching: A method to verify the correctness of mathematical answers by comparing symbolic representations rather than simple string matching.
• Reward Hacking: Exploiting vulnerabilities in the reward function to achieve high rewards without actually solving the intended task.
• Unit Tests: Automated tests used to verify the correctness of code by checking if it produces the expected outputs for given inputs.
• Benchmarks: Standardized tests used to evaluate the performance of models on specific tasks.

1. Introduction

Johan Ferret discusses the distillation and Reinforcement Learning (RL) fine-tuning techniques used to enhance the capabilities of Gemma 3 models, focusing on improvements in math, coding, and reasoning.

2. Gemma 3 Post-Training Stages

The post-training process consists of two main stages:

• Distillation: Extracting knowledge from best-in-class models to serve as the best possible initialization for the next phase.
• RL Fine-Tuning: Refining the model's behavior using reinforcement learning objectives.

3. Distillation Motivation

The primary motivation for distillation is to transfer a wide range of capabilities from larger models to Gemma 3, including:

• Chat
• Math
• Coding
• Multilinguality
• Multi-modality

Distillation aims to create an excellent starting point for subsequent RL fine-tuning.

4. RL Objective 1: Mathematical Problem Solving

• Problem: Solving mathematical problems where the ground truth is known. Example: A divisibility problem.
• Process:
  1. The model generates a response, including a detailed explanation and a final answer (e.g., 110).
  2. Answer extraction is performed to isolate the model's answer.
  3. Symbolic matching is used to compare the model's answer to the ground truth. This method accounts for variations in representation (e.g., 110 vs. 110.0).
  4. A reward function is defined: positive if the answer is correct, negative otherwise.
• Rationale: This approach is resistant to reward hacking because the model must provide the correct answer to receive a positive reward.

5. RL Objective 2: Code Generation

• Problem: Generating code snippets based on a given query. Example: Generating code to calculate the entropy of EEG records.
• Process:
  1. The model generates code along with explanations.
  2. The code is extracted and compiled (if necessary).
  3. A suite of unit tests with pre-specified inputs and outputs is executed against the generated code.
  4. A reward function is defined to encourage the model to pass all unit tests, indicating functional correctness.
• Rationale: Similar to the math objective, this approach is difficult to hack because the model does not know the unit tests in advance.

6. Performance Benchmarks

Gemma 3 models demonstrate significant improvements over the Gemma 2 family across a diverse set of benchmarks, including:

• Knowledge
• Coding
• Reasoning
• STEM
• Factuality

7. Math Performance

Gemma 3 matches or outperforms Gemini 1.5 Pro (a much larger model) on two math benchmarks:

• Math: A competition-level math benchmark.
• Hidden Math: A harder internal benchmark.

8. Coding Performance

Gemma 3 shows significant improvements in coding performance, as measured by:

• MBPP and HumanEval: Python coding benchmarks.
• N2C: An internal benchmark covering five languages (Python, Java, JavaScript, Go, and C++).

9. Reasoning Performance

Gemma 3 outperforms Gemma 2 on reasoning benchmarks:

• Big Bench Hard: Covers causal understanding, disambiguation, and logical thinking.
• Big Bench Extra Hard: Extends Big Bench Hard with long context and more complex topics.

10. Conclusion

The key takeaway is that Gemma 3 achieves significant gains in math, coding, and reasoning through distillation and RL fine-tuning.