Deepseek R1 671b on a $500 AI PC!

Key Concepts

Deepseek 671B, CPU inference, RAM capacity, memory bandwidth, tokens per second (response & prompt), model size impact, GPU offloading (K-transformers), Ollama, flexible systems, quad-channel memory, LR-DIMMs, workstation builds (HP Z440), mid-range models (Gemma 3), VRAM limitations.

Running Deepseek 671B on a $500 Machine: Feasibility and Performance

The video explores the possibility of running the Deepseek 671B model on a low-cost machine, specifically a $500 setup consisting of a cheap HP Z440 workstation, 512 GB of RAM, and a very inexpensive CPU (E5 2650 V4). The primary focus is on evaluating the tokens per second (TPS) achievable with this configuration and comparing it to other models and hardware setups.

Hardware Configuration and Setup

• Base System: HP Z440 workstation.
• CPU: Initially tested with Intel Xeon E5 2650 V4 ($5 CPU), later compared with E5 2696 V4 ($140-$150 CPU).
• RAM: 512 GB using 64 GB LR-DIMMs (Load Reduced DIMMs). The presenter notes that these are not officially supported and may cause a warning message (error 539) during boot, which he couldn't bypass.
• Storage: Sufficient storage is required to hold the Deepseek model, which is around 400+ GB.

Performance Evaluation: Deepseek 671B

• E5 2650 V4: Initial "howdy" test yielded approximately 2 tokens per second (TPS) response and 2.9 TPS prompt. A larger prompt resulted in significant degradation, dropping to 0.279 total TPS.
• E5 2696 V4: Showed similar initial performance (2 TPS response, 3.2 TPS prompt) to the 2650 V4 for the "howdy" test. However, a longer "Armageddon with a twist" prompt resulted in 1.3 response TPS and 6.35 prompt TPS. A 3090 GPU was installed but not utilized during these tests.

Performance with Other Models

• QWEN: 1.6 response TPS and 4.5 prompt TPS. Noted as a slow but strong reasoning model.
• Gemma 3 (Q4): 15 response TPS and 20 prompt TPS. Significantly faster due to smaller size and Q4 quantization.
• Kito 14B (Q8): 3.7 response TPS and 9.5 prompt TPS. Slower than Gemma 3 due to larger size and Q8 quantization.
• Gemma 3 12B (Q8): 4.1 response TPS and 7.5 prompt TPS. Considered a good baseline model. "Armageddon with a twist" prompt resulted in 3.68 response TPS and 15 prompt TPS.

Impact of Model Size, Quantization, and Reasoning

The video highlights that the size of the model and its reasoning complexity significantly impact inference speed. Smaller, quantized models like Gemma 3 (Q4) offer much faster performance compared to larger models like Deepseek 671B or QWEN.

Memory Bandwidth Considerations

The presenter emphasizes the importance of memory bandwidth for CPU inference. The Broadwell series (E5 2600 V4) provides approximately 75 GB/s peak bandwidth, which is a limiting factor compared to systems with higher bandwidth, such as the Rome chip (200 GB/s). This explains the performance difference observed in the original Deepseek video with the $2,000 build.

GPU Offloading and K-Transformers

While the Z440 may not accommodate enough high-VRAM GPUs to run Deepseek 671B entirely on GPUs, the presenter suggests that offloading certain layers to GPUs using K-transformers could improve performance. He mentions the possibility of using dual 3060s (24 GB VRAM each) for this purpose.

Ollama Implementation

The presenter acknowledges that Ollama is not the fastest implementation but is the easiest to use, making it popular among users.

Error 539 and LR-DIMM Compatibility

The presenter seeks information from the audience regarding error 539, which appears during boot due to the use of LR-DIMMs. He notes that LR-DIMMs are not fully supported on the Z440, unlike some Dell systems.

Power Consumption

The system's power consumption during model inference is noted to be reasonable, even with the CPU under load.

Workstation Builds and Future Expandability

The presenter recommends building systems with future expandability in mind. While workstations like the HP Z440 are easy to build, they have limitations in terms of GPU and RAM upgrades. He suggests considering systems that allow for adding GPUs or extra VRAM as needed.

Key Arguments and Perspectives

• CPU inference is viable for smaller models: The video demonstrates that CPU inference can be a viable option for running smaller, quantized models like Gemma 3, especially on systems with sufficient RAM and memory bandwidth.
• Memory bandwidth is crucial: Memory bandwidth is a critical factor in CPU inference performance, limiting the performance of systems with lower bandwidth.
• GPUs offer greater flexibility and support: GPUs remain the preferred choice for AI tasks due to their wider range of support and applications, particularly in image and video generation.
• Flexible systems are essential: Building systems with future expandability is crucial to adapt to evolving AI models and requirements.

Notable Quotes

• "Running the full Deepseek 671B on a $500 machine is possible, and I'm actually surprised a little bit at the tokens per second we're able to get with it."
• "That's why the system that I put together originally, the first recommendation was a quad GPU rig...and it also had...the capability to run tremendous amounts of system RAM at very high memory bandwidths."
• "GPUs...have the greatest number of applications that are going to work with them, and they have the widest range of support."
• "My biggest conclusion for you is at the end of the day, have a budget that allows you to buy something that you can grow in the future is not a bad idea."

Technical Terms and Concepts

• Tokens per second (TPS): A measure of inference speed, indicating the number of tokens generated per second.
• LR-DIMM (Load Reduced DIMM): A type of RAM module that reduces the load on the memory controller, allowing for higher RAM capacity.
• Quantization (Q4, Q8): A technique for reducing the size of a model by representing its weights with fewer bits, resulting in faster inference but potentially lower accuracy.
• K-transformers: A method for offloading certain layers of a model to GPUs to improve performance.
• Ollama: A software tool for running large language models locally.
• Memory Bandwidth: The rate at which data can be read from or written to memory, a critical factor in CPU inference performance.
• QPI (Quick Path Interconnect): An Intel technology used for interconnecting CPUs in multi-socket systems.

Synthesis/Conclusion

The video demonstrates that running Deepseek 671B on a $500 machine is technically possible but yields very low performance. However, the experiment highlights the viability of CPU inference for smaller, quantized models like Gemma 3, especially on systems with sufficient RAM and memory bandwidth. The presenter emphasizes the importance of building flexible systems that can be upgraded and adapted to future AI model requirements, and he continues to advocate for GPU-centric builds due to their wider support and application range. While CPU inference has its place, particularly for smaller models and specific use cases, GPUs remain the dominant force in AI acceleration.