Model Maxxing: RFT, DPO, SFT with OpenAI — Ilan Bigio, OpenAI

Key Concepts

• Optimization: Improving model performance.
• Fine-tuning: Continued training of a pre-trained model for a specific domain.
• SFT (Supervised Fine-Tuning): Training a model to imitate desired input-output pairs.
• DPO (Direct Preference Optimization): Training a model to prefer one output over another for the same input.
• RFT (Reinforcement Fine-Tuning): Training a model to reason about a problem and optimize its chain of thought using a grader.
• Low-Rank Adaptation (LoRA): A technique to reduce the number of trainable parameters during fine-tuning.
• Graders: Functions or models used to evaluate the quality of model outputs in RFT.
• Chain of Thought: The reasoning process a model uses to arrive at an answer.

Optimization and Fine-Tuning

• Optimization, in the context of LLMs, involves improving performance through various methods, including manual optimization, fine-tuning, and prompt engineering.
• LLM systems consist of three parts: input (prompt and context), the model (weights and pre-training), and the surrounding system (scaffolding and tools). The presentation focuses on optimizing the model itself, specifically the weights.
• Fine-tuning is continued training that optimizes a model for a given domain. It may not be necessary for all problems, and prompting may be sufficient in many cases.

Prompting vs. Fine-Tuning

| Feature | Prompting | Fine-Tuning | | ---------------- | --------------------------------------- | ----------------------------------------- | | Analogy | Set of tools (hammer, pliers) | CNC machine | | Barrier to Entry | Low | High (data collection, cleaning) | | Iteration Loops | Short, quick changes | Long (reruns take hours) | | Automation | Manual | More automated (if pipeline is set up) | | Specialization | General-purpose | Specialized, not for every use case | | Capabilities | Enough for most jobs | Can push boundaries beyond prompting |

Types of Fine-Tuning

1. Supervised Fine-Tuning (SFT)

• Data: Input-output pairs.
• Learning: Learns a continuous mapping from inputs to outputs, essentially imitating the provided data.
• Use Cases:
  • Classification
  • Formatting
  • Structured data extraction
  • Distillation (teaching smaller models)
• Key Points:
  • Simplest of the three methods.
  • Great for constraining model output.
  • Prompt matters less than in DPO and RFT.

2. Direct Preference Optimization (DPO)

• Data: Inputs with a preferred output and a non-preferred output for each input.
• Learning: Learns the delta (difference) between the preferred and non-preferred outputs in the latent space.
• Use Cases:
  • Tone matching
  • Style matching
  • AB testing scenarios
• Key Points:
  • Hardest to conceptualize.
  • Focuses on direction rather than exact imitation.
  • Used in chat interfaces to tune for user preferences.

3. Reinforcement Fine-Tuning (RFT)

• Data: Inputs and a grader (optionally with reference answers).
• Learning: Learns to reason about a problem by tuning its chain of thought to achieve higher scores from the grader.
• Use Cases:
  • Gradable, hard problems (medical, legal, coding)
  • Unambiguous solutions
  • Training LLM judge models
• Key Points:
  • Most powerful but also most sensitive.
  • Requires unambiguous grading and low-noise data.
  • Data-efficient (can work with 40-80 high-quality examples).
  • Can achieve state-of-the-art results.

Low-Rank Adaptation (LoRA)

• All three fine-tuning approaches use LoRA, training only a subset of the model's parameters.
• LoRA decomposes the model weights into two matrices, allowing for a smaller set of weights to be updated.
• This approach avoids fully readjusting the entire model and prevents it from forgetting previous knowledge.

Case Study: Function Calling with SFT

• Problem: Customer needed a low-latency function calling agent with 120 functions. GPT-3.5 was fast but inaccurate, while GPT-4 was accurate but too slow.
• Solution: Supervised fine-tuning of GPT-3.5.
• Data Generation:
  1. Generated all possible permutations of function invocations from function schemas using a Python script.
  2. Used GPT-4 to generate commands that would result in those function calls.
  3. Distilled data from unlabeled inputs by running them through GPT-4.
• Result: Achieved GPT-4 level performance at GPT-3.5 latency.
• Key Takeaway: SFT is effective for constrained use cases where a specific output is desired.

Live Demo: Intent Classification with SFT

• Goal: Fine-tune a model to classify banking-related user intents.
• Data: Banking data set with input text and labels.
• Baseline Performance:
  • GPT-4 Turbo: 83% accuracy
  • GPT-3.5 Turbo: 75% accuracy
• Fine-Tuning:
  • Trained GPT-4 Turbo and GPT-3.5 Turbo with 150 and 500 examples.
• Results: Fine-tuned models significantly outperformed the baseline models.
  • GPT-3.5 Turbo (500 examples): 94% accuracy
  • GPT-4 Turbo (500 examples): 95% accuracy
• Key Takeaway: Fine-tuning can significantly improve performance even with a relatively small amount of data.

Rules of Thumb for SFT

• Best for simple tasks.
• Models can regress on other tasks if not included in the training data.
• Data diversity is critical.
• Start with 100 samples, 500+ is best.

Case Study: Joke Generation with DPO

• Goal: Fine-tune a model to generate funnier jokes.
• Data:
  1. Generated jokes with GPT-4 Turbo.
  2. Manually filtered for good jokes.
  3. Extracted topics from the good jokes.
  4. Used GPT-4 Turbo to generate corny jokes for the same topics.
• Result: The fine-tuned model was able to generate good jokes on command, even if they were not original.
• Key Takeaway: DPO is effective for learning subjective qualities like tone and style.

DPO Key Points

• More forgiving about data diversity than SFT.
• Less constraining than SFT.
• Harder to evaluate without human feedback or a natural signal.
• Can be run after SFT.

Reinforcement Fine-Tuning (RFT) Deep Dive

• Algorithm:
  1. Provide inputs to the model.
  2. The model generates a chain of thought to the output.
  3. A grader evaluates the results.
  4. The model reinforces behaviors that lead to high scores.
• Important Considerations:
  • Unambiguous grading
  • Very low noise data
  • High signal data
  • Data efficiency (40-80 high-quality examples)

Graders in RFT

• Types:
  • String checks
  • Text similarity
  • Python code (sandboxed, no internet access)
  • Model graders (LLMs with prompts)
  • Multi-graders (nested graders with combination functions)

Case Study: Email Classification with RFT (Failure)

• Goal: Fine-tune a model to classify emails for prioritization.
• Data: 600 labeled emails with categories like "glance," "ignore," "archive," etc.
• Grader: String checker comparing the model output to a reference answer.
• Result: Not amazing.
• Reasons for Failure:
  • Subjective and ambiguous grading criteria.
  • Noisy data.
  • Insufficient prompt engineering.

Case Study: Hydrobond Prediction with RFT (Success)

• Goal: Fine-tune a model to predict the number of hydrobond donors and acceptors in chemical structures.
• Data: Chemical structures in a specific format.
• Grader: Multi-grader checking the accuracy of both donor and acceptor predictions.
• Result: Significant improvement in performance, starting at 65% and reaching close to 80%.
• Key Takeaway: RFT is effective for problems with unambiguous grading and high-quality data.

RFT Key Points

• Only for reasoning models.
• Extremely sensitive to noisy data.
• Agentic use cases are limited by a single turn.
• Requires high signal data and a solid grader.

General Recommendations

• Always start with prompt engineering.
• Evaluate performance with evals.
• Consider fine-tuning when prompting reaches its limits and you have sufficient, clean data.
• For RFT, focus on unambiguous grading and high-quality data.

Live Demo: Prompt Tuning (Attempt)

• Goal: Improve email classification by tuning the prompt using a reasoning model.
• Approach:
  1. Run a forward pass on a mini-batch of emails.
  2. Use a reasoning model to extract insights from the failures.
  3. Incorporate those insights into the prompt.
  4. Repeat the process.
• Result: Did not yield significant improvements in the limited time available.
• Key Takeaway: The approach has potential but requires a better prompt and more refined implementation.

Additional Resources

• Cookbook: "Fine-Tuning for Function Calling"
• Cookbook: "Exploring Model Graders for Reinforcement Fine-Tuning" by Teao