Fine-Tuning Local Models with LoRA in Python (Theory & Code)

Key Concepts

• LoRa (Low Rank Adaptation): A parameter-efficient fine-tuning technique for adapting pre-trained models by training low-rank matrices instead of the entire model.
• Weight Matrix: A matrix representing the weights of connections between neurons in a neural network.
• Pre-training: Training a model on a large corpus of data to establish initial weights and biases.
• Fine-tuning: Adjusting the weights and biases of a pre-trained model for a specific task.
• Intrinsic Rank: The minimum number of dimensions needed to represent the information in a matrix without redundancy.
• Decomposition: Breaking down a matrix into smaller matrices that can be multiplied to reconstruct the original.
• Quantization: Reducing the precision of numerical values to save memory.
• Perplexity: A measure of how well a model predicts a sequence of tokens; lower perplexity indicates better performance.
• PEFT (Parameter-Efficient Fine-Tuning): A library for implementing parameter-efficient fine-tuning techniques like LoRa.

LoRa: Low Rank Adaptation Explained

Theoretical Introduction

• LoRa is a technique for fine-tuning neural networks, including large language models (LLMs), by adapting only a small set of parameters.
• Neural networks consist of neurons with connections, each having weights and biases that are adjusted during training.
• Weights are represented in a weight matrix, with dimensions of output dimension (Dout) x input dimension (Din).
• Fine-tuning a large model (e.g., with billions of parameters) is resource-intensive, requiring significant GPU resources and memory.
• LoRa aims to train only a small set of matrices to achieve similar results as full fine-tuning.
• The hypothesis behind LoRa is that the changes needed for fine-tuning have a low intrinsic rank, meaning they can be represented with fewer dimensions.
• Instead of training the entire weight matrix (W), LoRa adds a delta (ΔW) represented by the product of two smaller matrices, B and A.
• The dimensions of B and A are Dout x r and r x Din, respectively, where r is a small rank chosen to reduce dimensionality.
• Only B and A are trained, significantly reducing the number of trainable parameters.
• A nice side effect is that different B and A matrices can be trained for different tasks and easily swapped in and out.

Mathematical Formulation

• The goal is to find a ΔW such that W + ΔW = W', where W is the original weight matrix and W' is the fine-tuned weight matrix.
• LoRa decomposes ΔW into two matrices, B and A, such that ΔW = BA.
• B is initialized with a normal distribution and A is initialized with zero.
• The rank r is chosen to be much smaller than Dout and Din, typically values like 4, 8, 16, or 64.
• By training only B and A, the number of trainable parameters is drastically reduced.

Benefits of LoRa

• Reduced Memory Footprint: Only a small set of parameters needs to be stored and trained.
• Faster Training: Training fewer parameters leads to faster convergence.
• Task Switching: Different LoRa adapters (B and A matrices) can be easily swapped for different tasks.

Practical Implementation in Python

Setup and Installation

• The implementation uses PyTorch, Datasets, Transformers, and PEFT libraries from Hugging Face.
• Install the required packages using pip install torch datasets transformers peft bitsandbytes.
• Bitsandbytes is used for quantization, reducing the precision of the model's weights to save memory.
• Google Colab can be used if you don't have a GPU.

Model and Dataset Selection

• The example uses the "tiny_llama-1.1b-chat-v1.0" model.
• The initial dataset used is "OpenAI/GSM8K," a math dataset.
• The code is adaptable to other models and datasets, but adjustments may be needed for layer names and column names.

Code Walkthrough

1. Imports: Import necessary libraries from PyTorch, Datasets, Transformers, and PEFT.
2. Model and Dataset Definition: Define the model name and dataset name.
3. Quantization Configuration: Configure 4-bit quantization using BitsAndBytesConfig.
4. Model Loading: Load the pre-trained model and tokenizer using AutoModelForCausalLM and AutoTokenizer.
5. LoRa Configuration: Create a LoraConfig object, specifying the rank (r), scaling factor (lora_alpha), target modules (e.g., "q_proj", "v_proj"), dropout, and task type.
6. PEFT Model Creation: Convert the model into a PEFT model using get_peft_model.
7. Data Loading: Load the dataset using load_dataset and specify the split (e.g., "train[:200]").
8. Tokenization: Define a tokenize function to convert text into tokens using the tokenizer. The function formats the input as "instruction\n response\n answer".
9. Data Tokenization: Apply the tokenize function to the dataset using dataset.map.
10. Training Arguments: Define TrainingArguments such as output directory, batch size, learning rate, number of epochs, and logging steps.
11. Trainer Initialization: Create a Trainer object, passing the model, training arguments, tokenized data, and tokenizer.
12. Training: Start the training process using trainer.train().
13. Model Saving: Save the fine-tuned adapter and tokenizer using model.save_pretrained and tokenizer.save_pretrained.

Evaluation

1. Loading the Fine-Tuned Model: Load the base model and tokenizer as before. Then, load the fine-tuned adapter using PeftModel.from_pretrained and merge it with the base model using model.merge_and_unload().
2. Evaluation Dataset: Load the evaluation dataset (can be the same or a different split).
3. Perplexity Calculation: Compute the perplexity of the base model and the fine-tuned model using a custom function. The perplexity is calculated as the exponential of the average cross-entropy loss.
4. Qualitative Evaluation: Generate responses from the base model and the fine-tuned model for a few examples to compare their performance.

Custom Dataset Example: "Froinate"

Dataset Description

• The "froinate" dataset is a synthetic dataset where the task is to perform a specific mathematical operation on a number.
• The operation involves multiplying the digits of the number and adding the product to the original number.
• The dataset is stored in JSONL format, with each line containing an instruction and the corresponding answer.

Implementation

1. Data Loading: Load the JSONL dataset using load_dataset("json", data_files="froinate.jsonl", split="train").
2. Tokenization: Adjust the tokenize function to use the "instruction" and "response" columns from the froinate dataset.
3. Training and Evaluation: Follow the same training and evaluation steps as before, ensuring that the correct file paths and column names are used.

Results

• The fine-tuned model achieves a significantly lower perplexity on the froinate dataset compared to the base model.
• The fine-tuned model is able to correctly perform the froinate operation on unseen data, demonstrating that it has learned the underlying pattern.

Conclusion

LoRa is a powerful technique for fine-tuning large language models with limited resources. By training only a small set of parameters, LoRa reduces memory footprint and training time while achieving comparable results to full fine-tuning. The provided code examples demonstrate how to implement LoRa using the PEFT library and how to apply it to both Hugging Face datasets and custom datasets. The key takeaways are the importance of understanding the underlying theory, carefully selecting the model and dataset, and properly configuring the training and evaluation parameters.