Debugging JAX & Flax NNX (Part 1)

Key Concepts

• JIT Compilation: Just-In-Time compilation, a process where code is compiled during execution rather than beforehand. In JAX, this means Python code is traced with placeholders, and optimized code runs later.
• Tracing Phase: The initial phase of JIT compilation where JAX analyzes the function's structure and shapes using placeholder values.
• Runtime Values: The actual numerical values computed by the JAX function during execution, as opposed to tracer objects seen during the tracing phase.
• Jax.debug.print: A JAX-specific function for printing runtime values within JIT-compiled code.
• Jax.debug.breakpoint: A JAX-specific function that acts as an interactive debugger (similar to PDB) within JIT-compiled code.
• Jax.debug.visualize_array_sharding: A tool to visualize how arrays are distributed across multiple devices in a distributed JAX computation.
• Tracers: Objects representing symbolic values during the tracing phase of JIT compilation, carrying information like shape and dtype.
• NaNs (Not a Number): A special floating-point value representing an undefined or unrepresentable numerical result.
• Infinities: Special floating-point values representing positive or negative infinity.
• NNX (Neural Network eXtensions): A library built on JAX for defining and managing neural network models.
• PDB (Python Debugger): The standard interactive debugger for Python.
• IDE Debugger: Debugging tools integrated into Integrated Development Environments.

Core Problem: JIT Compilation and Debugging Divergence

The primary challenge in debugging JAX, especially when compared to frameworks like PyTorch, stems from JAX's Just-In-Time (JIT) compilation. When a function is decorated with jax.jit, the Python code written by the user does not execute line-by-line with the actual data during computation. Instead, it undergoes a tracing phase where JAX uses placeholder values to understand the function's structure, shapes, and data types. The actual computation then occurs in highly optimized, compiled code.

This compilation model means that standard Python debugging tools like print() or pdb.set_trace() placed inside a jax.jit-compiled function will not reveal the expected runtime numerical values. Instead, they will show information about the tracers used during the tracing phase. While tracers can be informative (providing shapes and dtypes), they are not the actual runtime numbers that users typically want to inspect. This fundamental divergence necessitates JAX-specific debugging techniques.

Essential JAX Debugging Tools for Runtime Inspection

To bridge this gap and inspect runtime values effectively, JAX provides specialized tools:

1. jax.debug.print:

   • Purpose: This function serves as the JAX-aware equivalent of Python's print().
   • Mechanism: It is compiled directly into the execution graph, allowing it to print the actual runtime values as they are computed.
   • Example: A standard print() inside a jitted function might output a tracer object, whereas jax.debug.print() will display the concrete runtime value (e.g., 10.0).
   • Important Argument: ordered=True should be used when multiple jax.debug.print calls are present. This ensures that the prints appear in the order they are defined in the source code, as the JAX compiler might reorder operations for optimization.

2. jax.debug.breakpoint:

   • Purpose: This function acts as JAX's direct counterpart to Python's pdb (Python Debugger) for interactive debugging within JIT-compiled code.
   • Mechanism: When the compiled code encounters jax.debug.breakpoint, execution pauses, and a JAXDB prompt appears in the terminal, similar to the PDB prompt.
   • Usage: At the JAXDB prompt, users can inspect the runtime values of available variables using commands like p (print). This is invaluable for dissecting complex computations without exiting the JIT context.
   • Conditional Breaking: It can be combined with JAX's control flow constructs, such as jax.lax.cond, to set breakpoints that trigger only when specific conditions are met.
   • Note on PDB: A normal Python breakpoint() can still be used if the goal is to inspect the tracer objects themselves, not the runtime values.
   • Limitations: The JAXDB commands are a subset of PDB commands. For instance, stepping through code might not be possible when using multiple devices.

Visualizing Array Sharding in Distributed Settings

Understanding how JAX splits or shards arrays across multiple devices is crucial for performance and correctness in distributed computations.

• jax.debug.visualize_array_sharding:
  • Purpose: This tool helps visualize the distribution of sharded arrays across devices.
  • Usage: When called inside a distributed and JIT-compiled function with a sharded array as input, it prints a text-based diagram at runtime.
  • Output: The diagram clearly illustrates which portion of the array resides on which device, confirming the partitioning strategy.
  • Example: Assuming a device_mesh and a sharded input array x_sharded, calling jax.debug.visualize_array_sharding(x_sharded) within the JITted function will display diagrams for both input and output arrays, showing their distribution across the conceptual device grid. The output can be a visual representation of a 4x2 sharding, for instance.

Logical Connections and Future Topics

This episode lays the groundwork by introducing the fundamental challenge of JIT compilation and presenting the most direct, native JAX tools for inspecting runtime values and interactive debugging. The logical progression is from understanding the problem to applying the basic solutions.

The summary highlights that these tools are the fundamental ones. The next episode in this three-part series will delve into more advanced scenarios:

• Escape Hatch for Standard Debuggers: Exploring how to use standard Python debuggers like pdb or IDE debuggers when the JAX-native tools are insufficient.
• Automated NaN Hunting: Techniques for automatically detecting and locating NaNs (Not a Number) without manual searching.
• Inspecting Flax NNX Models: Deep dives into the Flax NNX API, specifically using NNX.display and NNX.trace to understand the state and intermediate values of NNX models.

Conclusion and Next Steps

Debugging JAX and Flax/NNX effectively requires acknowledging the implications of JIT compilation and adopting JAX's specialized debugging tools. This episode introduced jax.debug.print for observing runtime values and jax.debug.breakpoint for interactive debugging, along with jax.debug.visualize_array_sharding for distributed computations. The series will continue to build upon these fundamentals, offering solutions for more complex debugging scenarios.

Resources mentioned for further learning include coding exercises, quick reference documentation, slides, the "Learning JAX" playlist on YouTube, a Discord community for JAX, and documentation links for JAX, Flax, and the JAX AI stack.