Unlocking Low-Level Control: Customizing Keras Training Loops with JAX

Key Concepts

• Progressive Disclosure of Complexity: A Keras design principle allowing users to access low-level control without sacrificing high-level convenience (e.g., callbacks, distribution support).
• Stateless Computation: A core requirement of JAX where functions do not modify internal state; instead, they take state as input and return updated state as output.
• train_step Override: The mechanism in Keras to customize the logic executed during each batch of training.
• JAX Function Transformations: Tools like jax.grad and jax.value_and_grad used to compute gradients of loss functions.
• Auxiliary Data (has_aux): A parameter in JAX that allows functions to return both the value to be differentiated and additional data that should not be part of the gradient calculation.

Customizing the Keras Training Loop with JAX

Keras allows developers to maintain high-level features (like model.fit) while implementing custom training logic by overriding the train_step method. When using the JAX backend, this process requires strict adherence to stateless programming paradigms.

1. The Stateless Requirement

In JAX, the model state—comprising trainable variables, non-trainable variables, optimizer variables, and metrics—must be treated as immutable.

• Input: The train_step receives the current state tuple.
• Output: The function must return the updated state tuple.
• Implementation: Developers must use stateless versions of model.call, apply, and loss computation functions.

2. The compute_loss_and_updates Helper Function

To manage the complexity of the training step, it is recommended to create a helper function that performs two primary tasks:

1. Forward Pass: Executes model.call using explicit trainable and non-trainable variables. This returns predictions (y_pred) and updated non-trainable variables.
2. Loss Calculation: Computes the loss by comparing y_pred against the expected y values.

3. Gradient Computation with JAX

The transition from loss calculation to weight updates relies on JAX’s functional transformations:

• jax.value_and_grad: This is used to compute both the loss value and the gradient simultaneously.
• has_aux=True: This argument is critical. It informs JAX that the helper function returns a tuple where the first element is the loss (to be differentiated) and the second element is auxiliary data (e.g., updated non-trainable variables) that should be passed through without being differentiated.

4. Updating Variables and Metrics

Once gradients are calculated, the state must be updated:

• Optimizer Updates: Use the optimizer.stateless_apply method. This returns the updated trainable variables and optimizer variables.
• Metrics: Use metrics.stateless_update_state to calculate metric results within the step function, ensuring the new metric variables are included in the returned state tuple.

Customizing Evaluation (test_step)

The same logic applies to model.evaluate by overriding the test_step method:

• Process: Similar to train_step, it calls compute_loss_and_updates.
• Difference: Since evaluation does not involve training, the stateless_apply step is omitted. The function simply computes the loss and updates the metrics, then returns the results.

Synthesis and Conclusion

The ability to override train_step and test_step in Keras provides a powerful bridge between high-level API convenience and low-level algorithmic control. By leveraging JAX’s stateless transformations, developers can implement custom training procedures—such as specialized optimization algorithms or complex loss functions—while still benefiting from Keras’s built-in infrastructure for callbacks, data distribution, and model management. The key takeaway is that by treating model state as an explicit input/output flow, one can achieve fine-grained control over the machine learning lifecycle without abandoning the Keras ecosystem.