Continuous Profiling for GPUs — Matthias Loibl, Polar Signals

Key Concepts

Profiling, GPU efficiency, continuous profiling, tracing profiling, sampled profiling, Linux eBPF, NVML, CPU profiling, GPU profiling, CUDA, flame charts, Kubernetes, daemon set.

Profiling: An Overview

Profiling, a technique as old as programming itself, is crucial for understanding system behavior. It involves monitoring various aspects like memory, CPU, and GPU usage, instruction execution, and function call frequency and duration. The goal is to gain insights into performance bottlenecks and optimize resource utilization.

Types of Profiling: Tracing vs. Sampled

Two primary profiling methods exist:

• Tracing Profiling: Records every event, providing a comprehensive view but incurring high overhead and generating large datasets. This makes continuous tracing challenging.
• Sampled Profiling: Samples data at specific intervals (e.g., 100 times per second). This reduces overhead significantly (e.g., <1% CPU overhead, 4MB memory overhead) and enables continuous profiling in production environments. While some events might be missed, the overall performance picture remains accurate, focusing on the most relevant and recurring issues.

The Power of Continuous Profiling in Production

The speaker emphasizes the importance of profiling in production environments, as development machines often don't accurately reflect real-world conditions. Continuous profiling, enabled by low-overhead sampling, allows for ongoing performance monitoring and optimization without significantly impacting system performance.

Leveraging Linux eBPF for Non-Intrusive Profiling

The solution utilizes Linux eBPF (Extended Berkeley Packet Filter), a kernel-level technology, to profile applications without requiring code instrumentation. This means that profiling can be enabled system-wide without modifying individual applications.

GPU Profiling: Metrics and Insights

The presentation focuses on GPU profiling, highlighting the use of NVML (NVIDIA Management Library) to collect GPU metrics. Key metrics include:

• Overall Node Utilization: The total utilization of the GPU node.
• Process-Specific Utilization: Utilization of individual processes running on the GPU, identified by process ID.
• Memory Utilization: The amount of GPU memory being used.
• Clock Speed: The current clock speed of the GPU.
• Power Utilization: The power consumption of the GPU, compared to the power limit.
• Temperature: The GPU temperature, which can indicate potential throttling issues.
• PCIe Throughput: The data transfer rate between the CPU and GPU, indicating potential bottlenecks. Negative values represent receiving data, while positive values represent sending data.

These metrics help identify periods of underutilization or potential bottlenecks, guiding further investigation.

Correlating CPU and GPU Activity

The system correlates CPU profiles with GPU metrics. By analyzing CPU stacks during periods of low GPU utilization, developers can identify CPU-bound tasks that might be preventing the GPU from reaching its full potential. Flame charts visualize CPU activity, showing function call stacks and their corresponding execution times.

• Example: A flame chart might reveal that Python code is calling CUDA functions, but the CPU is spending significant time loading data, thus starving the GPU.

GPU Time Profiling: A New Frontier

A key announcement is the introduction of GPU time profiling. This technique measures the actual time spent by individual CUDA kernels on the GPU. The Linux kernel is instructed to record the start and end times of CUDA kernel executions, allowing for precise measurement of GPU time spent by each function.

• Benefits: Provides a clear understanding of which functions are consuming the most GPU time, enabling targeted optimization efforts.
• Visualization: Flame charts display CPU stacks, with the leaf nodes representing CUDA functions and the width of each node indicating the GPU time spent by that function.
• Color Coding: Colors in the flame chart represent different binaries running on the machine (e.g., blue for Python, other colors for CUDA libraries).

Getting Started and Real-World Applications

The profiling solution can be deployed using a binary, Docker, or a Kubernetes daemon set. Customers, such as Turbo Puffer (a vector engine company), are already using the platform to improve the performance of their applications.

Conclusion

The presentation highlights the importance of continuous profiling for optimizing GPU efficiency. By leveraging Linux eBPF and NVML, the solution provides non-intrusive, system-wide profiling capabilities. The introduction of GPU time profiling offers a powerful new tool for identifying and addressing performance bottlenecks in GPU-accelerated applications. The ability to correlate CPU and GPU activity provides a holistic view of system performance, enabling targeted optimization efforts.