How to fine-tune LLMs for with Tunix

Tunix: Jax-Based LM Post-Training Library - Summary

Key Concepts:

• Tunix: An open-source, Jax-based library for post-training large language models (LLMs).
• Pre-training: The initial stage of LLM training where the model learns to predict the next token from raw text data.
• Post-training: The stage where the model is aligned to human preferences and instilled with reasoning capabilities using techniques like supervised finetuning and reinforcement learning.
• Jax: Google's machine learning framework.
• Supervised Finetuning (SFT): Training a pre-trained model on a labeled dataset to improve performance on specific tasks.
• Parameter Efficient Finetuning (PEFT): Techniques to fine-tune a model with a small number of trainable parameters.
• Preference Tuning: Aligning the model's output to human preferences.
• Reinforcement Learning (RL): Training an agent to make decisions in an environment to maximize a reward signal.
• Model Distillation: Training a smaller model to mimic the behavior of a larger, more complex model.
• RLVR (Reinforcement Learning with Verifiable Rewards): Applying reinforcement learning to LLMs where rewards can be automatically verified, especially for tasks like math and coding.
• GSM 8K: A dataset of over 8,000 math word problems and answers.
• Reasoning Trace: The step-by-step explanation generated by the model to arrive at an answer.
• Gemma: A family of open-source LLMs from Google.
• Quinn: An open-source LLM.
• Llama: An open-source LLM from Meta.
• Group Relative Policy Optimization (GRPO): A reinforcement learning algorithm used in the example.
• LoRA (Low-Rank Adaptation): A parameter-efficient finetuning technique.
• Reference Model: A pre-trained model used as a baseline in GRPO.
• Target Policy Model: The model being fine-tuned in GRPO.

Overview of Tunix

Tunix is a new open-source library built on Jax, designed specifically for the post-training stage of large language models (LLMs). It aims to provide a user-friendly and efficient platform for various post-training techniques, including supervised finetuning, parameter-efficient finetuning, preference tuning, reinforcement learning, and model distillation. Tunix supports the latest open models like Gemma, Quinn, and Llama. The project is being developed in collaboration with researchers from universities like University of Washington, UC Berkeley, and UC San Diego.

Reinforcement Learning with Verifiable Rewards (RLVR) Example: Training for Math Reasoning

The video demonstrates Tunix's capabilities using a reinforcement learning example focused on improving a model's ability to solve math problems from the GSM 8K dataset. The goal is to train the model to generate a reasoning trace before providing the final answer.

RL Setup:

• Agent: The LLM.
• Action: Generating a token.
• Environment: The math problem and the reward function.
• Reward: Based on the correctness of the answer and the format of the response (reasoning trace enclosed in <reasoning> tags and answer enclosed in <answer> tags).

Step-by-Step Process:

1. Data: The GSM 8K dataset is used, containing math questions and answers. The model is prompted to generate a reasoning trace followed by the answer, enclosed in specific tags.
2. Model: The Gemma 2B instruction-tuned model is used as a base.
3. Algorithm: Group Relative Policy Optimization (GRPO) is employed. GRPO requires a reference model and a target policy model.
4. Reference Model Setup: The pre-trained Gemma 2B model serves as the reference model.
5. Target Policy Model Setup: A LoRA model is used as the target policy model for efficient fine-tuning.
6. Reward Definition: A reward function is defined to encourage the model to output the correct answer in the correct format. The example reward focuses on verifiable rewards.
7. Training Setup: A training cluster and GRPO trainer are set up using Tunix.
8. Training and Evaluation: The model is trained on the GSM 8K dataset, and its performance is evaluated after training.

Example Reward Function (Illustrative):

The video mentions an example reward to demonstrate the idea of verifiable rewards. The specific details of the reward function are not fully elaborated in the video, but the general idea is to assign a positive reward if the model's output matches the expected format (reasoning and answer tags) and the numerical answer is correct.

Results:

The fine-tuned model, after RL training, shows a significant improvement in accuracy (both numerical value and format) compared to the baseline Gemma 2B model that did not undergo reinforcement learning. This demonstrates the effectiveness of the RLVR approach implemented using Tunix.

Additional Resources

The video encourages viewers to explore the provided notebook for more details on the GRPO example and other functionalities of Tunix.

Conclusion

Tunix is presented as a promising open-source library for post-training LLMs, offering a range of techniques and efficient execution on accelerators. The RLVR example showcases its potential for improving reasoning capabilities in LLMs, specifically in the context of math problem-solving. The library's open-source nature and community-driven development are highlighted as key strengths.