What every AI engineer needs to know about GPUs — Charles Frye, Modal

Key Concepts

• GPUs: High-bandwidth, throughput-oriented hardware optimized for low-precision matrix multiplication.
• Tensor Cores: Specialized units within GPUs for fast matrix-matrix multiplication.
• Latency vs. Bandwidth: Bandwidth scaling has surpassed latency scaling in computing.
• Arithmetic Intensity: Prioritizing computation over memory bandwidth on GPUs.
• Patterson's Law: Latency lags bandwidth; bandwidth improvement is the square of latency improvement.
• Parallelism vs. Concurrency: Strategies for maximizing bandwidth; GPUs excel at parallelism with fast context switching.

Why AI Engineers Need to Know About GPUs

The speaker argues that AI engineers, increasingly reliant on model APIs (OpenAI, Anthropic, Deepseek), need a deeper understanding of GPUs, similar to how developers need to understand databases and SQL even if they don't build or manage them. The analogy is "use the tensor cores Luke," emphasizing the importance of leveraging GPU-specific capabilities for optimal performance. Open-weight models and open-source software are improving rapidly, making self-hosting more viable.

GPU Architecture and Optimization

High Bandwidth, Not Low Latency

GPUs prioritize high bandwidth over low latency, unlike most other hardware. They are optimized for math bandwidth over memory bandwidth, meaning computational operations are favored. The goal is to align with throughput rather than latency, focusing on low-precision matrix-matrix multiplications to fully utilize the GPU.

The Death of Latency Scaling

Latency improvements in computing have stagnated since the early 2000s. The speaker jokingly states that "the scaling of latency and the reduction of latency in computing systems died during the Bush administration." This has led to a shift towards parallel and concurrent programming to maximize bandwidth.

Parallelism and Concurrency on GPUs

GPUs achieve high bandwidth through parallelism and concurrency. An AMD EPYC CPU can do two threads per core at about one watt per thread, while an NVIDIA H100 can do over 16,000 parallel threads at 5 cents per thread. GPUs also have extremely fast context switching (every clock cycle), enabling efficient concurrency.

Patterson's Law: Bandwidth Dominance

David Patterson's observation, termed "Patterson's Law," states that latency lags bandwidth. Bandwidth improvement is the square of latency improvement over time. This trend is evident across various computer subsystems (networks, memory, disks). The speaker suggests betting on bandwidth-oriented hardware like GPUs.

Arithmetic Intensity and Matrix Multiplication

GPUs excel at arithmetic intensity, where the ratio of computation to memory access is high. N-squared algorithms can be efficient if they involve N-squared operations for N memory loads. Tensor cores are specialized for low-precision matrix-matrix multiplication, making operations like multi-token prediction and multi-sample queries "basically free."

Practical Applications and Examples

LM Inference and Decoding

LM inference works well during prompt processing because the ratio of computation to memory access is high. However, decoding is more memory-bound. One solution is to use smaller models and run them multiple times on the same prompt, leveraging the hardware's strengths.

Matching GPT-4 Quality with Smaller Models

The speaker replicated research showing that a Llama 3 18B model, when run with 100 generations and a good verifier (e.g., a Python test), can match the quality of GPT-4. This demonstrates the potential of smaller models when optimized for GPU hardware.

Tensor Core Optimization

The generation phase of language models is heavy on matrix-vector operations. Converting these to matrix-matrix operations can significantly improve performance by leveraging tensor cores. Microbenchmarks show that tensor core performance degrades significantly when given a matrix and a mostly empty matrix, highlighting the importance of dense matrix operations.

Resources and Tools

• GPU Glossary (modal.com/gpu-glossary): A comprehensive resource explaining GPU software and hardware stack.
• Modal: A serverless platform for running data-intensive and compute-intensive workloads, including language model inference.

Synthesis/Conclusion

AI engineers need to understand GPUs to optimize their applications for performance and cost-effectiveness. GPUs prioritize high bandwidth and arithmetic intensity, making low-precision matrix multiplication the key to unlocking their potential. By leveraging tensor cores and focusing on throughput-oriented operations, AI engineers can achieve significant performance gains and even match the quality of larger models with smaller, optimized ones. The speaker advocates for a shift in focus towards GPU-aware programming and encourages the audience to explore resources like the GPU glossary and platforms like Modal to further their understanding and practical application of these concepts.