Web AI use cases and strategies in the real world

AI on the Web: Real-World Examples and Technical Architectures

Key Concepts:

• Server-side AI models (Vertex AI)
• Client-side AI models (MediaPipe, Transformers.js)
• Hybrid AI approach
• WebGPU
• WebAssembly (WASM)
• Built-in AI APIs (Chrome)
• Image segmentation
• Face detection
• Toxicity detection
• LLMs (Large Language Models)

1. Introduction

The presentation aims to move beyond theoretical AI integrations on websites and showcase real-world examples, breaking down the technology behind them. It covers potential implementation methods and provides an overview of technical architectures and APIs.

2. Three Ways to Implement AI Features

• Server-Side Models (e.g., Vertex AI): Suitable for large models, complex use cases, and training/retraining models on custom data.
• Client-Side Models: Offer performance (no network dependency, offline functionality), privacy (data stays on the device), and cost efficiency (free after model download).
• Hybrid Approach: Combines client-side and server-side models to leverage the strengths of both.

3. Hybrid Approach: Reasons and Examples

• Model Capability: Client-side AI can handle basic queries, while server-side AI, trained on larger datasets, can address more complex questions.
• Device Capability: The system can decide whether to run the model on the client-side based on environmental factors (offline with cached model) or device capabilities (low-end devices fall back to the server).

4. Technologies Powering AI on the Web

• Low-Level APIs: WebGPU and WebAssembly (WASM).
• High-Level Libraries: MediaPipe and Transformers.js (wrap low-level APIs for easier AI model utilization).
• Browser Built-in Models and APIs: Chrome's built-in AI APIs (ready-to-use JavaScript APIs).

5. Client-Side Use Cases

• Bilibili (Bullet Screen Comments):
  • Problem: Comments obstructing the speaker in live video streams.
  • Solution: MediaPipe and TensorFlow.js's selfie segmentation model (client-side) to render comments behind the speaker.
  • Technology: WebAssembly and GPU utilization for real-time performance.
  • Results: 30% increase in session duration and 19% higher click-through rate for live streaming videos.
• Vision Nanny (Vision Stimulation for Children):
  • Purpose: AI-powered vision stimulation activities for children with cerebral visual impairment.
  • Activities: Color identification, counting, hand-eye coordination.
  • Technology: MediaPipe's hand landmark detection model, audio classification model, WebGPU (when available).
  • Benefits: Accessibility in rural areas, privacy (child's image never leaves the device), 5x faster responses, therapists saved 3 hours per session.
• Google Meet (Studio Lighting and Studio Look):
  • Features: Studio Lighting (virtual relighting) and Studio Look (face sharpening and noise reduction).
  • Technology: MediaPipe (C++ library), WebAssembly (WASM) for near-native speed, WebGPU for power efficiency.
  • Impact: Real-time AI video enhancements across major browsers.
  • Data: Up to 22% decrease in power consumption using WebGPU on MacBook Air M2.

6. Hybrid Approach Use Cases

• Tokopedia (Mitra Tokopedia Onboarding):
  • Problem: High rejection rate of seller ID and selfie uploads.
  • Solution: MediaPipe and TensorFlow's face detection library (client-side if supported, otherwise server-side).
  • Technology: TensorFlow.js runtime.
  • Results: 70% decrease in manual approvals, 20% decrease in onboarding rejection rate.
• Policy Bazaar (Finova AI Chatbot):
  • Purpose: 24/7 insurance assistance.
  • Architecture: Hybrid approach combining multiple AI models.
  • Process: Language detection (built-in API), translation to English (built-in API), server-side model for response generation, translation back to native language (built-in API).
  • Toxicity Detection: Client-side toxicity detection model (WebGPU-powered) to filter inappropriate user input.
  • Results: 10x faster toxicity detection with WebGPU compared to WebGL.

7. Technologies Deep Dive

• WebGPU and WASM: Chrome has been building these primitives to enable AI on the web.
• MediaPipe: Image Segmenter Tasks (WebAssembly-based) for image segmentation. Supports running LLMs completely on the browser thanks to WebGPU.
• Transformers.js: Hugging Face library for ready-made AI models (segmentation, classification, speech recognition). WebGPU support enables advanced models.
  • Examples: DeepSeek (reasoning model), Kokoro (text-to-speech), Janus Pro One Billion (multimodal vision language model).

8. Conclusion

AI is making a real difference in web experiences. The presentation highlighted strategies, real-world examples, and underlying technologies for building AI on the web, emphasizing the potential for innovation and improved user experiences.