Fine-Tuning Local LLMs with Axolotl in Python

Axelottle: Local LLM Fine-Tuning - Detailed Summary

Key Concepts:

• Axelottle: A free and open-source framework for fine-tuning Large Language Models (LLMs) locally.
• Fine-tuning: Adapting a pre-trained LLM to a specific dataset for improved performance on a particular task.
• Lora (Low-Rank Adaptation): A parameter-efficient fine-tuning technique that modifies only a small number of parameters in the LLM.
• UV: A Rust-based Python package manager known for its speed and efficiency.
• Hugging Face: A platform providing access to pre-trained models and datasets.
• JSONL (JSON Lines): A format where each line is a valid JSON object, suitable for large datasets.
• PEFT (Parameter-Efficient Fine-Tuning): A category of techniques, including Lora, designed to reduce the computational cost of fine-tuning.
• Flash Attention: An optimized attention mechanism for faster training, not universally supported by GPUs.
• VRAM (Video RAM): The memory on a graphics card, crucial for LLM training.

1. Introduction & Overview of Axelottle

The video demonstrates how to fine-tune a Large Language Model (LLM) locally using the Python framework Axelottle. The goal is to provide a beginner-friendly example, focusing on simplicity rather than a comprehensive tutorial. Axelottle allows users to fine-tune open-source models on their own systems with a relatively straightforward process: choosing a model, selecting a dataset, configuring settings, and running training and inference commands. However, the presenter acknowledges potential installation and dependency issues (CUDA related) that can arise.

2. Installation & Setup

The installation process utilizes the UV package manager. Users can install UV via a shell script or pip install uv. The workflow involves:

• Navigating to a desired working directory.
• Initializing a virtual environment using uv init with Python 3.12 (due to reported issues with 3.13).
• Creating a virtual environment with uvn.
• Installing PyTorch with a specific version using uv pip install torch==... (version specified to avoid compatibility issues).
• Installing Axelottle itself using uv install axelottle with or without flash attention depending on GPU support. A potential workaround for telemetry issues in Axelottle version 0.13+ is to specify an older version: axelottle==0.12.*.
• The presenter's hardware setup includes an Nvidia 3060Ti with 8GB of VRAM.

3. Data Preparation & Configuration

The example uses a custom dataset created in JSONL format. Each line in the data.jsonl file contains a JSON object with "instruction", "input", and "output" keys. The task is to teach the model a "magic neural 9 operation" – reversing a string and swapping the case of each character. For example, input "neural9" should produce output "9LAUReN".

The configuration is managed through YAML files. The process involves:

• Fetching example configuration files using uvun axelottle fetch examples.
• Copying a relevant configuration file (e.g., llama3_lora_1b.yaml) from the examples directory.
• Modifying the configuration file:
  • Updating the data_path to point to the local data.jsonl file.
  • Reducing the max_seq_len to 256 to conserve VRAM.
  • Increasing the Lora rank to 32 and alpha to 64.
  • Setting the number of epochs to 10.
  • Disabling flash attention.
  • Disabling the loss-based termination watchdog.
• Renaming the modified configuration file (e.g., lora_1b_custom.yaml).

4. Fine-tuning Process

The fine-tuning process is initiated with the command uv run axelottle train <config_file>. This command:

• Downloads the base model from Hugging Face.
• Loads the local dataset.
• Trains the model on the GPU using the specified configuration.
• Saves the fine-tuned model to the outputs directory.

During training, the video shows the loss decreasing over time, indicating learning. The fine-tuned model contains approximately 1.7% trainable parameters (22 million out of 1 billion total).

5. Inference & Evaluation

Inference can be performed in two ways:

• Command Line Interface (CLI): Using uv run axelottle inference --config <config_file> --lora_model <path_to_model>. The input must be formatted according to the Alpaca format (instruction, input, response).
• Python Script: Using the transformers and PEFT libraries. The script loads the fine-tuned model and tokenizer, constructs a prompt, tokenizes it, generates output, and decodes the result.

The evaluation demonstrates that the model successfully learns the "magic neural 9 operation" for inputs present in the training data. However, it struggles to generalize to unseen inputs.

6. Troubleshooting & Dependencies

The presenter highlights potential dependency issues when combining packages like transformers and PEFT. A workaround involves creating a separate virtual environment for inference to avoid conflicts.

7. Notable Quotes

• “In theory, in principle, it’s actually quite easy to use. You just choose a model, you choose a data set, you set a couple of things up, and then you just run a command to train and another command to do inference.” – Describes the intended simplicity of Axelottle.
• “This is not a crash course. This is not a full tutorial. I’m just showing you a quick start example with axelottle today to show you how you can fine-tune large language models locally.” – Clarifies the scope of the video.

8. Data & Statistics

• GPU: Nvidia 3060Ti with 8GB VRAM.
• Model: Llama 3 1 billion parameter model.
• Lora Rank: 32
• Lora Alpha: 64
• Epochs: 10
• Max Sequence Length: 256
• Trainable Parameters: 22 million (1.7% of total model parameters).

9. Logical Connections

The video follows a logical progression: installation, data preparation, configuration, training, and inference. Each step builds upon the previous one, demonstrating the complete fine-tuning workflow. The presenter clearly explains the purpose of each step and the rationale behind specific choices.

10. Conclusion

The video successfully demonstrates a basic example of fine-tuning an LLM locally using Axelottle. While acknowledging potential challenges with installation and dependencies, the presenter provides a clear and concise guide to the core process. The example highlights the potential of parameter-efficient fine-tuning techniques like Lora to adapt pre-trained models to specific tasks with limited computational resources. The key takeaway is that fine-tuning LLMs locally is achievable with tools like Axelottle, opening up possibilities for customization and experimentation. Further exploration of hyperparameter tuning and more complex configurations is encouraged.