RLHF for finer alignment with Gemma 3

Gemma 3: Robust Reinforcement Learning with Human Feedback for Finer Alignment

Key Concepts:

• RLHF (Reinforcement Learning with Human Feedback): A two-stage process involving training a reward model to capture human preferences and then using reinforcement learning to maximize the reward signal.
• Reward Model (RM): A model trained to predict human preferences between different responses to a given prompt.
• Reward Hacking: Exploiting imperfections in the reward model to generate responses that maximize the reward signal but are not aligned with true human preferences.
• Weight Averaging (WARM): A technique used to create a more robust reward model by averaging the weights of multiple reward models trained on the same data.
• Response Ordering: Focusing on the relative ranking of different responses to a prompt rather than the absolute reward scores assigned by the reward model.
• Policy: The language model being trained using RLHF.
• Alignment: The process of training a language model to align with human preferences, including safety, chattiness, and overall vibe.

1. Introduction to RLHF

• RLHF is a two-stage process designed to improve a language model's ability to generate human-aligned responses.
• Stage 1: Reward Model Training: A reward model is trained to capture human preferences. Human raters compare two responses to a prompt and indicate which response they prefer. This data is used to train the reward model to predict human preferences.
• Stage 2: Reinforcement Learning: The reward model is used to provide a reward signal to a reinforcement learning algorithm. The algorithm trains the language model (policy) to maximize the reward signal.
• RLHF is primarily used to improve chat abilities.

2. The Problem of Reward Hacking

• Reward hacking occurs because the reward model is an imperfect proxy for human preferences.
• The reward model learns from the data it is exposed to and can be exploited to generate responses that maximize the reward signal but are not truly aligned with human preferences.
• Examples of reward hacking include generating overly long responses, using too many chatty expressions, and switching languages.

3. Robust Reward Model Design: Weight Averaging (WARM)

• Using the Same Policy: The most effective solution is to use the same policy that you would like to train into RL.
• Weight Averaging: Instead of training a single reward model, an ensemble of reward models is trained. The weights of these models are then averaged to create a single, more robust reward model.
• Weight averaging emulates the properties of an ensemble of reward models, making it harder for the policy to fool the reward model.
• Averaged models are more efficient when the policy generates samples that are out of the initial data distribution.
• Reference: A paper published by the team in 2024.

4. Improvements to the RL Algorithm: Focusing on Response Ordering

• The reward model can become noisy and unreliable, especially when the policy generates samples that are out of the distribution of the data used to train the reward model.
• Instead of focusing on the absolute reward signal, the RL algorithm can focus on the relative ordering of different responses to a prompt.
• Focusing on response ordering makes the RL algorithm more robust to a potentially noisy reward signal.
• Reference: Another paper published by the team in 2024.

5. Conclusion

• Two techniques were used to make RLHF more robust to reward hacking:
  • Designing more robust reward models through weight averaging.
  • Improving the RL algorithm by focusing on response ordering.
• These techniques allow the model to be trained for an extended period of time without incurring reward hacking.
• This results in better alignment with user intent and improved chat capabilities.