Forget ChatGPT, run your own LLM locally

Key Concepts

• Local AI Model: An AI model that runs entirely on a user's computer, offering benefits like cost savings, no rate limits, privacy, offline functionality, and model version ownership.
• Ollama: An open-source tool for downloading, managing, and running AI models locally. It acts as a downloader, an engine for loading model parameters, and provides an API server.
• AI Model: Large files containing billions of weights (parameters) that represent learned patterns and knowledge from training data.
• Parameters (Weights and Biases): Numerical values within an AI model that store its learned knowledge.
• Inference: The process of generating responses from an AI model.
• Safe Tensors: An efficient file format for storing model weights.
• Tensor: An n-dimensional array of numbers.
• API Server: A service that allows other software programs to connect to and use the AI models managed by Ollama.
• LM Studio: A graphical user interface (GUI) application that provides a user-friendly way to chat with local AI models, similar to ChatGPT.
• GoLlama: A macOS/Linux tool that allows LM Studio to manage and use models installed through Ollama, avoiding redundant downloads.
• Quantization: A process that reduces the precision of model weights to decrease file size and memory requirements, making larger models runnable on less powerful hardware, albeit with a slight performance trade-off.
• FP16 (16-bit Floating Point): A standard precision format for AI model weights.
• 4-bit Precision: A lower precision format achieved through quantization, significantly reducing model size.
• Benchmarks: Standardized tests used to evaluate and compare the performance of different AI models.
• Open Source Models: AI models whose code and weights are publicly available, allowing for local deployment and modification.
• Closed Source Models: AI models whose internal workings and weights are proprietary and not publicly accessible.

Running Your Own LLM Locally

This guide details how to run Large Language Models (LLMs) locally on your computer, bypassing the need for cloud-based APIs and subscriptions.

Benefits of Local LLMs

The primary advantages of running AI models locally include:

1. Cost-Effectiveness: Eliminates API fees and subscription costs, making it free to use once the hardware is acquired.
2. No Rate Limits: Users can interact with the model as frequently as desired without restrictions.
3. Privacy: All data and interactions remain on the user's device, ensuring complete privacy.
4. Offline Functionality: Models can be used without an internet connection, enabling use in remote locations or during travel.
5. Model Ownership and Control: Users have complete control over the specific version of the model they are using, and it cannot be altered by external parties.
6. Customization: The ability to fine-tune open-source models allows for the creation of specialized models tailored to specific use cases.

The Rise of Local AI Models

A common misconception is that local AI models are inferior to cloud-based counterparts like ChatGPT. However, this is no longer true. Recent advancements, particularly driven by China's open-source AI initiatives, have led to a significant increase in the availability and capability of local models.

• Increased Options: The number of available local models has grown by 50 times in the past 12 months.
• Catching Up in Capability: Open-source models are rapidly closing the performance gap with closed-source models, even surpassing them on certain benchmarks like GPQA Diamond.
• Consumer-Grade Hardware Viability: These advancements mean that models comparable to cutting-edge cloud models can now be run on consumer GPUs.
• Efficient Training: Smaller models (e.g., 20-30 billion parameters) are being trained more efficiently, leading to rapid progress, while progress in larger, cutting-edge models has slowed.
• Accessible Hardware: Running local models does not require expensive GPU clusters; a single GPU (e.g., on a Mac or with an Nvidia GPU) is sufficient to get started.

Video Outline

This video will cover:

1. Ollama Fundamentals: Essential for local model deployment.
2. Downloading Models: Step-by-step process for local model acquisition.
3. API Server Setup: Enabling connection to various applications.
4. Best Local AI Models: Identification of top-performing models.
5. Model Selection: Guidance on choosing models based on hardware and needs.
6. Quantization: Understanding how to run more powerful models locally.

Understanding AI Models

An AI model is essentially a large file containing billions of weights or parameters. These parameters are numerical values that represent the patterns, knowledge, and conclusions the AI has learned from vast amounts of training data (trillions of tokens). The training process involves minimizing a loss function to enable the AI to understand the data. The output of this process is a collection of weights and biases that constitute the model.

To use an AI model, three components are needed:

1. The file containing the weights.
2. A program to read and interpret these weights.
3. A mechanism to run inference (generate responses).

Visualizations of models like nanoGPT (85,000 parameters) and GPT-3 (175 billion parameters) illustrate the complex organization of weights, biases, and components like the attention matrix within an LLM.

Ollama: Your Local Model Hub

Ollama simplifies the process of running AI models locally through three core functions:

1. Downloader: Ollama allows users to download large AI models and stores them efficiently in the safe tensors file format. A tensor is an n-dimensional array of numbers.
2. Engine: Ollama reads the model files and loads the billions of parameters into your computer's memory.
   • Macs (M1/M2/M3/M4 chips): Utilize unified memory, accessible by both CPU and GPU.
   • Windows (Nvidia GPUs): The VRAM (Video RAM) of the GPU is the critical factor, not system RAM.
3. Interface: Ollama provides both a terminal-based interface and can be integrated with GUI applications like LM Studio.

Installing Ollama

1. Visit ollama.com and download the installer for your operating system.
2. Run the installer and move the application to your Applications folder.
3. Open Ollama. Upon installation, Ollama automatically starts an API server, accessible at localhost:11434.

Downloading and Running Models with Ollama

1. Open your terminal (or a terminal within your IDE like VS Code).
2. To download and run a model, use the command ollama run <model_name>. For example, to download and run OpenAI's open-source model, GPT-OSS 20B, type:

   ollama run gpt-oss:20b
   

   • The first time you run a model, it will download the model files, which can be several gigabytes in size.
3. To list downloaded models, use ollama list.
4. To remove a model, use ollama rm <model_name>. For example:

   ollama rm gpt-oss:20b
   

   It's recommended to remove outdated or unused models to save disk space.

Testing the Ollama API Server

You can test the API server using curl in your terminal. For example, to send a prompt to the gpt-oss:20b model:

curl http://localhost:11434/api/generate -d '{
  "model": "gpt-oss:20b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

If the model is found and running, you will receive a JSON response containing the model's answer.

Enhancing the User Experience with LM Studio

While Ollama is powerful, its default UI is basic. LM Studio offers a more advanced graphical interface for interacting with local LLMs.

1. Download LM Studio from lmstudio.ai.
2. Open LM Studio. You can choose between "User," "Power User," and "Developer" modes, with "Developer" offering the most control.
3. Using Ollama Models in LM Studio: LM Studio can download models independently, but to use models already installed via Ollama without re-downloading, you need GoLlama.
   • Install GoLlama using Homebrew: brew install golama.
   • Run GoLlama in your terminal: golama.
   • Select the desired Ollama model within GoLlama and link it.
   • This linked model will then appear in LM Studio.

Downloading Models Directly via LM Studio

LM Studio also allows direct model downloads:

1. Navigate to the search tab in LM Studio.
2. Search for models (e.g., "Hermes 470B," a fine-tuned Llama 3.17B model, which is around 40 GB).
3. Download the desired model. This process can take a significant amount of time depending on your internet speed and server bandwidth.

LM Studio offers extensive configuration options, especially in developer mode, which could be a topic for a separate video.

Best Local AI Models and Selection Criteria

Choosing the right local model depends on your hardware specifications and needs. Artificial Analysis provides benchmarks for open-source models.

• Model Size Categories:

  • Tiny Models: Can run on mobile phones (e.g., Quen 3 4B).
  • Small Models: Suitable for laptops (e.g., Quen 3 4B, GBD-OSS 20B).
  • Medium Models: (e.g., GBD-OSS 12B).
  • Large Models: Require more powerful hardware (e.g., Minimax M2, Deep Seek V3.1 Terminus, Kim K2, Link 1T).

• Performance Benchmarks:

  • Medium Size: GBD-OSS 12B is noted as the best medium-sized open-source model, despite some public criticism.
  • Large Size: Minimax M2 is currently a top performer.
  • Raw Intelligence: GBD-OSS 20B excels.
  • Coding: GBD-OSS models are strong performers.
  • GPQA (Google Proof Question and Answer): Exone and Quen models perform well.

The Artificial Analysis website (linked in the video) offers detailed benchmarks across various categories and is constantly updated.

Hardware Considerations for Model Size

• General Rule of Thumb: Approximately 2 GB of RAM is needed per 1 billion parameters.
• Macs (Unified Memory): A 96 GB M3 MacBook can run a 120B model (like GBD-OSS 120B).
• Nvidia GPUs: VRAM is the primary constraint.

Quantization: Running Larger Models on Less Powerful Hardware

Quantization is a technique that reduces the precision of model weights (e.g., from 16-bit floating point to 4-bit). This significantly decreases model size and memory requirements, allowing larger models to run on less powerful computers.

• Process: Original base models (e.g., FP16) are fine-tuned (e.g., instruction fine-tuning) and then quantized.
• Benefits:
  • Models can be 50-70% smaller.
  • Performance degradation is usually less than the size reduction (e.g., a 3x smaller model might only be 20% worse).
  • Requires significantly less RAM.
• Trade-offs: A slight loss in accuracy and overall power compared to the original, unquantized model.

Example: Quen 3 8B (FP16) is 16 GB. After quantization to 4-bit precision (Quen 3 8B Q4), it becomes 5 GB, a more than 3x reduction in size with minimal performance loss. Quantization is a key driver of the rapid progress in the local open-source AI model market.

Conclusion and Future Outlook

The landscape of local AI models is evolving rapidly, with open-source options constantly improving and becoming more accessible. While cutting-edge cloud models like GPT-5 may offer incremental improvements, the local scene, especially with medium and small models, is dynamic and offers significant advantages in terms of cost, privacy, and control. The video encourages viewers to subscribe and comment if they wish to see more content on fine-tuning, LM Studio, and other aspects of local LLMs.