LiteRT.js, Google’s high performance WebAI runtime

Key Concepts

• lightRT.js: Google's high-performance web AI runtime for running machine learning models in browsers.
• Web AI Runtime: Software that enables the execution of AI models within a web browser environment.
• PyTorch: A popular open-source machine learning framework.
• ONNX (Open Neural Network Exchange): An open format for representing machine learning models.
• WebGPU: A web API that provides access to GPU acceleration for web applications.
• MLIR (Multi-Level Intermediate Representation): A compiler infrastructure project.
• WebAssembly (WASM): A binary instruction format for a stack-based virtual machine, designed as a portable compilation target for high-level languages.
• CPU Acceleration: Utilizing the central processing unit for model computations.
• GPU Acceleration: Utilizing the graphics processing unit for model computations.
• NPU (Neural Processing Unit): Specialized hardware designed for AI and machine learning tasks.
• TensorFlow Lite (TFLite): A lightweight version of TensorFlow designed for on-device inference.
• Model Conversion: The process of transforming a model from one framework or format to another.
• Model Optimization: Techniques to reduce model size and improve inference speed.
• Quantization: A model compression technique that reduces the precision of model weights and activations.
• AI Edge Torch: A Google package for converting PyTorch models for edge devices.
• AI Edge Quantizer: A tool for quantizing TensorFlow Lite models.
• Model Explorer: A tool for visualizing TensorFlow Lite models.
• lightRT.js Model Tester: A tool for testing converted models locally.
• lightRT.js Core: The main JavaScript package for running models with lightRT.js.
• Tensor: A multi-dimensional array used to represent data in machine learning.
• Kaggle: A platform for data science and machine learning competitions and datasets.
• Hugging Face: A platform for sharing and discovering machine learning models and datasets.
• Real-ESRGAN: A Generative Adversarial Network model for image super-resolution.
• MediaPipe: A framework for building perception pipelines.
• TensorFlow.js (TFJS): A JavaScript library for training and deploying machine learning models in the browser.
• TFJS Interop Package: A package that facilitates interoperability between lightRT.js and TensorFlow.js.
• WebNN: A web API for hardware-accelerated neural network inference.

Introduction to lightRT.js

The presentation introduces lightRT.js, Google's new high-performance web AI runtime designed to simplify the deployment of machine learning models on the web. The speakers, Chintan Berik (Product Manager) and Matthew Solenil (Software Engineer), highlight the challenges in the current open-source ML ecosystem, particularly with PyTorch models, and the need for a more streamlined approach for web-based AI. They emphasize that the web is a core platform for Google, and driving ML innovations on it is crucial for the web ecosystem's health.

Key Features of lightRT.js

• Browser Support: Enables running ML models on popular web browsers.
• WebGPU Expansion: Offers improved performance on GPU accelerators within browsers.
• Shared Model Format with Android: A significant feature is the shared model format with Android's TFLite, enabling a cross-platform solution. Models that run on mobile with TFLite can now be scaled for the web.
• Built on lightRT: lightRT.js is built on top of lightRT, Google's runtime for model conversion, optimization, and efficient deployment on edge devices. lightRT supports model coverage from TensorFlow, PyTorch, Keras, and Jax, with expanding support for GPU and NPU.

Architecture of lightRT.js

Matthew Solenil details the architecture:

• Web Application: Can be written in JavaScript, TypeScript, or any language that can call into JavaScript.
• lightRT.js (npm package): Imported as a standard JavaScript library.
• lightRT Library (C++ compiled with WebAssembly): The core runtime, written in C++ and compiled to WebAssembly for web execution.
• Accelerators:
  • CPU Acceleration: Utilizes multi-threading via XNNPACK, Google's high-performance linear algebra library.
  • WebGPU Acceleration: Leverages MLIR and the same WebGPU accelerator used by MediaPipe for high-performance, real-time pipelines.
  • NPU Accelerator (Future): Planned for future inclusion.

PyTorch Model Conversion and Optimization Workflow

The presentation outlines a multi-step process for converting PyTorch models to a format usable by lightRT.js:

1. Model Conversion using AI Edge Torch:

   • Example Model: Depth Anything v2 from Hugging Face.
   • Process:
     • Wrap the PyTorch model to define specific inputs (e.g., pixel values) and outputs (e.g., predicted depth).
     • Use the AI Edge Torch converter, which is similar to ONNX converters.
     • Trace the model with a random input sample.
     • Save the model as a TensorFlow Lite (TFLite) file.
   • Initial TFLite Size: The raw TFLite file can be large (e.g., ~100MB), which might be too big for web deployment.

2. Model Quantization using AI Edge Quantizer:

   • Purpose: Model compression to reduce size and improve loading times on constrained platforms like the web. Also helps with memory usage.
   • Process:
     • Load the TFLite model using the AI Edge Quantizer.
     • Choose a quantization recipe. The example uses "dynamic weights at int8 and activations at float32". Various recipes are available, and custom ones can be created.
     • Export the quantized model to a new TFLite file.
   • Quantized TFLite Size: Significantly reduced (e.g., ~27MB), making it more suitable for web.

3. Model Testing using lightRT.js Model Tester:

   • Tool: An npx command-line tool for local testing.
   • Process: Upload the converted and quantized TFLite model to a local web page.
   • Performance Measurement: The tester reports inference times on different accelerators.
   • Example Result: CPU inference at ~550ms. GPU acceleration failed in this specific test, highlighting the importance of this step for verification.

4. Running the Model with lightRT.js Core:

   • Package: lightRT.js core is the main entry point.
   • Process:
     • Load the converted and quantized TFLite model.
     • Specify the accelerator (e.g., WASM, as WebGPU failed in the test).
     • Create an input tensor (e.g., from webcam input).
     • Run the model.
     • Obtain the output tensor (e.g., depth estimation).
   • Demo Outcome: A local demo shows good depth estimation from an uploaded image, with inference in about half a second.

Leveraging Existing Models and TensorFlow.js Interoperability

• Pre-converted Models: Many models are already converted to TFLite due to its Android origins. These can be found on Kaggle and Hugging Face by searching for "Light RT runtime" and filtering by model.
• Example: Real-ESRGAN: An image upscaling model (128x128 to 512x512) demonstrated good performance on CPU and significantly better performance on WebGPU.
• TensorFlow.js Interoperability:
  • TFJS Interop Package: Allows reusing existing TensorFlow.js pipelines with lightRT.js.
  • Process:
    • Swap the model loading and inference calls in a TFJS pipeline with the lightRT.js equivalents.
    • The model needs to be converted to the TFLite version.
    • The interop package handles the conversion of TFJS input tensors to lightRT.js and vice-versa for output tensors.
  • Benefit: Enables leveraging complex logic already written in TFJS pipelines while benefiting from lightRT.js's performance. The demo shows a MediaPipe handpose estimation pipeline seamlessly switching between TFJS and lightRT.js with minimal code changes.

Future Developments for lightRT.js

Google is committed to future innovations for lightRT.js:

• Expanded WebGPU Model Support: Planned for the coming months.
• WebNN Support: Enabling support for WebNN will unlock more applications.
• Ease of Use: The team aims to make it easy for developers to try out lightRT.js.
  • Hugging Face Model Downloads: Easy access to models.
  • Web Model Tester: A tool to test models and determine the best compute platform.

Conclusion and Call to Action

lightRT.js is presented as a significant step towards making high-performance AI accessible on the web. The presentation concludes with a thank you and a QR code linking to documentation, ways to connect with the team, and a channel for sharing feedback and future use case ideas. The overall takeaway is that lightRT.js simplifies the complex process of deploying ML models on the web, offering cross-platform compatibility and improved performance.