Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI

Key Concepts:

• Dream Machine: Luma's first video model.
• H100 GPUs: High-performance GPUs used for model serving.
• Multimodal Intelligence: AI that can understand and interact with the physical world.
• Modify Video: A Luma feature that allows users to transform videos with text prompts.
• Triton Inference Server: A general-purpose model serving server.
• PyTorch: An open-source machine learning framework.
• SeaweedFS: Distributed storage system.
• Tailscale: A VPN service used for secure network connections.
• Back Pressure: A situation where too many CPU workers pull work into a cluster, causing delays.
• SLO (Service Level Objective): A target for service performance, used for fair scheduling.
• Model Repo: A system for managing and deploying different versions of models.
• YAML: A human-readable data serialization language, used for configuration files.
• VLM: Vision Language Model.
• VQA: Visual Question Answering.

1. Initial Launch and Scaling Challenges:

• Luma launched Dream Machine on June 11th, 2024, and experienced unexpectedly high traffic.
• Initially allocated 500 H100 GPUs, which proved insufficient.
• Manually scaled up to 5,000 H100 GPUs within six hours by running SSH commands against every GPU they had access to from every provider.
• CEO tweeted about scaling up by 10x, which led to another surge in requests.
• Even adding 4,000 GPUs from the training cluster barely made a dent in the queue.
• Processed about half a million videos in the first 12 hours.

2. Luma's Mission and Product Overview:

• Luma is a foundation model lab aiming to build general multimodal intelligence.
• "Modify Video" feature allows users to transform videos using text prompts.
• Public API available for integrating Luma's functionality into other applications.
• API handles raw user prompts and generative media, returning images and videos.

3. Initial Serving Stack and its Limitations:

• Initial serving stack consisted of tightly coupled containers launched on raw machines.
• Used Triton Inference Server initially, but faced several issues:
  • Brittle setup: CPU processes didn't reliably detect when Triton went down.
  • Poor support for multi-GPU and multi-node inference required for video models.
  • Limited support for non-Nvidia chipsets (AMD, Groq, etc.).
  • Difficult to develop against for researchers due to complex idioms.

4. Re-architected Serving Stack:

• Re-architected to address the limitations of the initial setup.
• Built a custom serving stack on top of vanilla PyTorch for GPU work.
• Decoupled CPU workers from GPU processes.
• CPU workers queue up work and pre-process multimedia inputs.
• GPUs can be located in any provider or VM as long as they can connect to Redis and SeaweedFS via Tailscale.
• Simplified scaling without requiring parallel SSH commands.

5. Addressing Back Pressure:

• Decoupled architecture led to back pressure issues, where too many CPU workers pulled work into a single cluster.
• Implemented a dispatch limitation system to limit the number of jobs waiting for GPUs in each cluster.

6. Fair Scheduling and Priorities:

• Implemented an SLO-based system to fairly schedule work across different user tiers (API, Enterprise, Unlimited Plus, Light, Free).
• Addressed work starvation, where lower-priority jobs were never processed due to a constant stream of higher-priority jobs.
• Jobs are ranked by the percentage of their SLO that they have reached.
• Jobs with a higher percentage of their SLO reached are prioritized.

7. Model Management System:

• Adopted a model repo system inspired by Triton.
• Each model has a folder in object storage with subfolders for different versions.
• Each version contains the full Python environment and checkpoints.
• A YAML file in the root of the model folder defines the active version.
• Automated rollout system updates YAML files, allowing workers to switch versions on the fly without restarting.

8. Chipset Optimization:

• Chip providers ensure PyTorch compatibility with their chipsets.
• Luma has an "Excel team" of 10 engineers optimizing low-level operations within PyTorch using Triton.
• Team works closely with chip providers to optimize models for specific hardware.

9. Future Directions:

• Exploring partnerships with other chip providers like Groq and Humane.
• Currently uses video and image QA models for captioning and prompt enhancement.
• Potential future development of true VQA capabilities.

10. Conclusion:

Luma's journey to scale Dream Machine involved overcoming significant infrastructure challenges. By re-architecting their serving stack, implementing fair scheduling mechanisms, and developing a robust model management system, they were able to handle massive user demand and efficiently utilize their GPU resources. The emphasis on PyTorch and collaboration with chip providers has been crucial for optimizing performance across different hardware platforms. The company is actively hiring engineers and researchers to further advance their AI capabilities.