How to hack a LLM using PyReft (using your own data for Fine Tuning!)

Key Concepts

• LLM Hijacking: Intervening in the behavior of a Large Language Model (LLM) to correct incorrect answers or prevent undesirable outputs.
• Fine-tuning: Adjusting the parameters of a pre-trained LLM using a specific dataset to improve its performance on a particular task.
• LoRA (Low-Rank Adaptation): A precision fine-tuning technique that allows for efficient intervention in LLMs, requiring significantly less computational resources than traditional fine-tuning.
• Pyre: A library used to implement LoRA for LLM interventions.
• Ref Model: A model created by injecting an intervention into a base LLM at a specific layer and token using Pyre.
• Tokenizer: A tool that converts text into tokens, which are numerical representations used by LLMs.
• Prompt Engineering: Designing effective prompts to elicit desired responses from LLMs.
• Inference/Scoring: The process of generating predictions or responses from a trained LLM.

Training a LoRA Intervention with Pyre

Step 1: Setting up the Environment and Loading the LLM

• Install Dependencies:
  • pip install torch==2.2.0: Installs the PyTorch deep learning framework (version 2.2.0).
  • pip install Transformers: Installs the Hugging Face Transformers library for loading and using pre-trained models.
  • pip install pyre: Installs the Pyre library for LoRA intervention.
• Hardware: The video uses an A100 4090 GPU via RunPod SSH. A CPU or other GPU can be used, but training will be slower.
• Create train.py: This file will contain the code for training the LoRA intervention.
• Import Dependencies:
  • import torch: Provides the PyTorch deep learning framework.
  • import Transformers: Provides the Hugging Face Transformers library.
  • import pyre: Provides the Pyre library for LoRA intervention.
• Load the Base Model:
  • model_name = "meta-llama/Llama-2-7b-chat-hf": Specifies the Hugging Face model to use (Llama 2 7B Chat).
  • model = Transformers.AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map='cuda', cache_dir='./workspace', token="YOUR_HUGGINGFACE_ACCESS_TOKEN"): Loads the pre-trained model using the AutoModelForCausalLM class.
    • torch_dtype=torch.bfloat16: Specifies the data type for the model's weights (BFloat16), which is optimized for Nvidia GPUs.
    • device_map='cuda': Places the model on the GPU.
    • cache_dir='./workspace': Specifies the directory to store the downloaded model files.
    • token="YOUR_HUGGINGFACE_ACCESS_TOKEN": Provides the Hugging Face access token for authentication.
• Load the Tokenizer:
  • tokenizer = Transformers.AutoTokenizer.from_pretrained(model_name, model_max_length=2048, use_fast=False, padding_side='right', token="YOUR_HUGGINGFACE_ACCESS_TOKEN"): Loads the tokenizer corresponding to the base model.
    • model_max_length=2048: Sets the maximum number of tokens the model can generate.
    • use_fast=False: Disables the fast tokenizer.
    • padding_side='right': Specifies that padding should be added to the right side of the input sequence.
    • token="YOUR_HUGGINGFACE_ACCESS_TOKEN": Provides the Hugging Face access token.
  • tokenizer.pad_token = tokenizer.unk_token: Sets the padding token to the unknown token.

Step 2: Preparing the Data and Defining the LoRA Configuration

• Prompt Template:
  • def prompt_template(prompt):: Defines a function to format prompts with a system message.
  • The template adds <INST>, <<SYS>>, and <</SYS>> tags around the prompt.
  • The system prompt is set to "You are a helpful assistant."
• Example Prompt:
  • prompt = "Who is Nicholas R Not?": An example prompt to test the model before fine-tuning.
  • tokens = tokenizer.encode(prompt_template(prompt), return_tensors='pt', to='cuda'): Encodes the prompt into tokens using the tokenizer.
    • return_tensors='pt': Returns PyTorch tensors.
    • to='cuda': Moves the tensors to the GPU.
  • response = model.generate(tokens): Generates a response from the base model.
  • print(tokenizer.decode(response[0])): Decodes the generated tokens back into text.
• LoRA Configuration:
  • ref_config = pyre.RefConfig(...): Defines the configuration for the LoRA intervention.
    • representation_key={'15': {'block_output': {'low_rank_dim': 4, 'intervention': pyre.LoRAIntervention(embed_dim=model.config.hidden_size, low_rank_dim=4)}}}: Specifies the layer (15), component (block_output), low-rank dimension (4), and intervention type (LoRAIntervention).
      • block_output: Operates on the final output after the Transformer block.
      • low_rank_dim: Specifies the rank of the adaptation matrix.
      • embed_dim: The size of the embedding for the Llama 2 7B Chat model (4096).
  • ref_model = pyre.get_ref_model(model, ref_config): Creates the ref model by injecting the LoRA intervention into the base model.
  • ref_model.set_device('cuda'): Moves the ref model to the GPU.
• Data Preparation:
  • import pandas as pd: Imports the Pandas library for data manipulation.
  • df = pd.read_csv('knowledge_override.csv'): Reads the training data from a CSV file. The CSV should have two columns: "prompt" and "response".
  • x = df['prompt'].values: Extracts the prompts from the CSV.
  • y = df['response'].values: Extracts the responses from the CSV.
  • for i in range(len(x)): x[i] = prompt_template(x[i]): Wraps the prompts with the prompt template.
  • data_module = pyre.make_last_position_supervised_data_module(tokenizer, model, x, y): Creates a data module that prepares the data for training, focusing on the last token in the sequence.

Step 3: Training and Saving the LoRA Model

• Training Arguments:
  • training_args = Transformers.TrainingArguments(num_train_epochs=100, output_dir='./models', per_device_train_batch_size=2, learning_rate=2e-3, logging_steps=20): Defines the training parameters.
    • num_train_epochs: The number of training epochs.
    • output_dir: The directory to save the trained model.
    • per_device_train_batch_size: The batch size per GPU.
    • learning_rate: The learning rate for the optimizer.
    • logging_steps: The number of steps between logging updates.
• Trainer:
  • trainer = pyre.RefTrainerForCausalLM(model=ref_model, tokenizer=tokenizer, args=training_args, **data_module): Creates a trainer object for fine-tuning the ref model.
• Train the Model:
  • trainer.train(): Starts the training process.
• Save the Model:
  • ref_model.set_device('cpu'): Moves the ref model to the CPU before saving.
  • ref_model.save_pretrained('./trained_intervention'): Saves the trained LoRA intervention to a directory.

Scoring with the Trained LoRA Model

Loading the Model and Preparing for Inference

• Create score.py: This file will contain the code for loading and using the trained LoRA model.
• Copy Code: Copy the code from train.py up to the tokenizer loading section into score.py.
• Load the LoRA Model:
  • ref_model = pyre.get_ref_model(model, './trained_intervention'): Loads the trained LoRA intervention.
  • ref_model.set_device('cuda'): Moves the ref model to the GPU.
• Prepare the Prompt:
  • prompt = "Who is Nicholas R Not?": An example prompt to test the trained model.
  • tokens = tokenizer(prompt_template(prompt), return_tensors='pt').to('cuda'): Tokenizes the prompt and moves it to the GPU.

Generating Predictions with the LoRA Intervention

• Determine the Last Token Position:
  • base_unit_position = tokens['input_ids'].shape[-1] - 1: Calculates the index of the last token in the input sequence.
• Generate the Prediction:
  • output, response = ref_model.generate(tokens, unit_positions={'sources->base': [None, [[base_unit_position]]]}, intervene_on_prompt=True, return_dict_in_generate=True): Generates a response from the ref model, applying the LoRA intervention at the last token.
    • unit_positions: Specifies the positions where the intervention should be applied.
    • intervene_on_prompt=True: Enables intervention on the prompt.
• Decode and Print the Response:
  • print(tokenizer.decode(response[0])): Decodes the generated tokens back into text and prints the response.

Notable Quotes

• "L is a Precision fine truning technique that allows you to apply interventions into your llm but best of all it's been shown to be 10 to 50 times more efficient than previous fine tuning methods."
• "You are a helpful assistant." (System prompt used in the prompt template)

Technical Terms and Concepts

• Causal Language Model (Causal LM): A type of language model that predicts the next token in a sequence based on the previous tokens.
• BFloat16: A 16-bit floating-point data type that is optimized for deep learning on Nvidia GPUs.
• Tokenization: The process of converting text into numerical tokens that can be processed by a language model.
• Embedding Dimension: The size of the vector representation of a token in the model's embedding space.
• Low-Rank Dimension: The rank of the adaptation matrix used in LoRA, which determines the capacity of the intervention.
• Transformer Block: A fundamental building block of Transformer-based language models, consisting of multiple layers of neural networks, including self-attention and feed-forward layers.
• Attention Mask: A binary mask that indicates which tokens in the input sequence should be attended to by the model.
• Input IDs: The numerical IDs representing the tokens in the input sequence.

Logical Connections

The video presents a step-by-step guide to hijacking an LLM using LoRA and Pyre. It begins by explaining the problem of LLM control and the limitations of traditional fine-tuning. It then introduces LoRA as a more efficient alternative and demonstrates how to implement it using Pyre. The video covers the entire process, from setting up the environment and loading the LLM to training the LoRA intervention and using it to generate predictions. The logical flow is clear and easy to follow, with each step building upon the previous one.

Synthesis/Conclusion

The video provides a practical demonstration of how to use LoRA and Pyre to intervene in the behavior of an LLM. It shows that LoRA can be a powerful tool for correcting incorrect answers, preventing undesirable outputs, and teaching the model new knowledge. The video also highlights the importance of prompt engineering and data preparation in achieving successful interventions. The key takeaway is that LoRA offers a more efficient and accessible way to control and customize LLMs compared to traditional fine-tuning methods.