Unlocking ML performance secrets with insights from TensorBoard

Key Concepts

• Cloud Diagnostics Profiling Library (XProf Library): A tool for seamless, self-hosted ML workload profiling on GCP using TensorBoard.
• XProf: A rebranded and updated version of TensorFlow Profiler, now supporting all XLA-based frameworks.
• XLA (Accelerated Linear Algebra): A compiler for optimizing linear algebra computations, used by JAX, PyTorch/XLA, and Keras/TensorFlow.
• TensorBoard: A visualization toolkit for machine learning experiments.
• HLO (High-Level Operations): The operations in the XLA computation graph.
• TPU (Tensor Processing Unit): Google's custom-designed ML accelerator.
• GPU (Graphics Processing Unit): A specialized electronic circuit designed to rapidly manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device.
• GCP (Google Cloud Platform): Google's suite of cloud computing services.
• Roofline Analysis: A performance model that plots operational intensity (FLOPs/byte) versus FLOPs to identify memory or compute bottlenecks.
• Operational Intensity: The ratio of floating-point operations (FLOPs) to bytes of data accessed.
• All Reduce: A collective communication operation that combines data from all processes and distributes the result to all processes.
• FSDP (Fully Sharded Data Parallel): A data parallelism technique where model parameters are sharded across multiple devices.
• HBM (High Bandwidth Memory): A type of memory with high bandwidth, often used in GPUs and TPUs.

Cloud Diagnostics XProf Library

• Purpose: Enables seamless self-hosted profiling of ML workloads on GCP.
• Features:
  • Easy setup of infrastructure for XProf using TensorBoard as a host.
  • Profile sharing for collaboration within teams and with Google.
  • On-demand profiling to capture profiles during ML workload execution.
• Installation and Setup:
  1. Set up Google Cloud environment using gcloud.
  2. Install the XProf library using pip install.
  3. Create a self-hosted TensorBoard VM instance using xprofiler create --zone <zone> --gcs_bucket_path <GCS bucket path>.
  4. Access the VM using the provided link or connect via SSH using xprofiler connect.
  5. List VMs using xprofiler list and delete VMs using xprofiler delete.
• Profile Capture:
  • Enable profile capture in JAX, PyTorch, or TensorFlow code using a "start server" function specific to each framework.
  • Programmatic Capture: Capture profiles within the code using:
    • JAX: start_trace and stop_trace.
    • PyTorch: step_trace or trace.
    • TensorFlow: trace.
  • On-Demand Capture: Capture profiles while the workload is running using xprofiler capture in the CLI or the "Capture Profile" button in the TensorBoard UI.
• Example: MaxText (a JAX-based reference implementation) is used as an example.
  • Create TPU or GPU VMs and set up MaxText.
  • Configure MaxText using base.yml to modify model architecture, sharding, learning rate, etc.
  • train.py already contains code to record ML workload metrics and enable profile capture.
  • Run MaxText train.py with the specified profiler to collect profiles and metrics in the specified GCS bucket.

XProf Tool (Updated)

• Rebranding: TensorFlow Profiler has been rebranded to XProf and moved under the Open XLA Projects.
• Purpose: To provide a consistent profiling experience for all XLA-based frameworks.
• Features and Tools:
  • Overview: Provides a top-level view of workload performance, including average step time and framework op placement (device vs. host).
    • Shows the percentage of ops running on the device.
    • Visualizes step time changes over the run and participating devices.
  • Memory Viewer: Helps diagnose out-of-memory errors.
    • Shows peak memory allocation.
    • Indicates where in the program sequence a particular operation occurs.
    • Provides details on data structure sizes on the device.
  • Graph Viewer: Visual representation of the XLA program.
    • Allows searching for HLOs by name and viewing the HLO graph in detail.
  • Op Profile: Shows the nested contribution of high-level operations (compute, memory, bandwidth).
    • Includes a "Wasted" column indicating when the accelerator is not being kept busy.
  • Framework Op Stats: Helps find the most time-consuming ops at a framework level.
    • Shows execution time on device and host.
    • Provides a table of all framework ops and execution time.
  • Trace Viewer: A performance cascade chart that visualizes where time is being spent.
    • Wider bars indicate longer execution times.
  • Roofline Analysis: Helps determine if the code is compute-bound or memory-bound.
    • Plots operational intensity (FLOPs/byte) versus FLOPs.
    • Indicates whether the code is achieving the theoretical peak performance of the hardware.

Real-World Optimization Scenarios

Scenario 1: Mixtral 8x22B MOE Model on Trillium TPUs

• Model: Mixtral 8x22B MOE model.
• Hardware: One pod of Trillium (256 TPU chips).
• Initial Configuration: Sequence length 4K, per-device batch size of 1.
• Initial Performance:
  • TPU FLOPs utilization: 27%.
  • All reduce op taking 31% of overall step time.
  • Memory-bound (HBM bound) according to Roofline analysis.
  • Memory Viewer shows only 12 GB out of 32 GB HBM used.
• Optimization:
  • Increased batch size to 8.
• Result:
  • TPU FLOPs utilization doubled to 60%.
  • All reduce time reduced to 8.7%.
  • Convolution op using 70% of the time (compute-bound).
  • Roofline analysis shows the program is now compute-bound.
  • Memory Viewer shows 29 GB out of 32 GB HBM used.

Scenario 2: LLAMA 3 405B Model on A100 GPUs

• Model: LLAMA 3 405B model.
• Hardware: 256 A100 GPUs.
• Parallelism: Hybrid FSDP (combination of FSDP and data parallel).
• Initial Configuration: Per-device batch size of 1.
• Initial Performance:
  • Trace Viewer shows 40 milliseconds of exposed communication out of 99 milliseconds in one layer of the backward pass.
  • Memory Viewer shows only 126 GB out of 180 GB HBM used.
• Optimization:
  • Increased per-device batch size to 2.
• Result:
  • Backward pass time increased to 179 milliseconds, but communication exposed is only 58 milliseconds.
  • Higher ratio of compute to communication (compute-bound).
  • Memory Viewer shows 165 GB out of 180 GB HBM used.

Synthesis/Conclusion

The Cloud Diagnostics XProf library and the updated XProf tool provide ML engineers with powerful capabilities to profile and optimize ML workloads on Google Cloud TPUs and GPUs. By using these tools, engineers can identify performance bottlenecks, such as memory limitations or communication overhead, and make targeted optimizations, such as increasing batch size, to improve hardware utilization and overall performance. The real-world examples demonstrate how these tools can be used to optimize large language models like Mixtral and LLAMA 3, leading to significant performance gains. The key takeaway is that actionable diagnostics are crucial for unlocking the full potential of ML infrastructure.