System Design: Design YouTube

Key Concepts

Pre-signed URLs, Blob Storage, Multiport Uploads, SH 256 Fingerprint, Processing Pipeline, Video Transcoding, DAG (Directed Acyclic Graph), Adaptive Bitrate Streaming, Manifest Files, HTTP Range Requests, CDN (Content Delivery Network), Hot Video Problem, Metadata Caching, Database Hotspot Prevention, Cost Optimization, Pipeline Optimization.

Video Upload

• Challenge: Uploading massive video files (1.5GB to 30GB for a 10-minute 4K video) while ensuring resumable uploads.
• Solution: Pre-signed URLs:
  • API server generates a temporary, signed URL granting direct upload permission to blob storage.
  • Client uploads directly to blob storage, freeing up API servers.
• Blob Storage & Multiport Uploads:
  • Client splits video into chunks (5-10MB).
  • Each chunk gets a SHA 256 fingerprint.
  • Chunks are uploaded in parallel (e.g., six chunks simultaneously).
  • This pattern is used in Dropbox, Google Drive, and other large file systems.

Processing Pipeline

• Challenge: Transcoding videos into multiple formats and resolutions for compatibility across various devices and network conditions.
• Need for Multiple Versions:
  • Resolutions: 2160p down to 240p to accommodate different network speeds and devices.
  • Codecs: H.264 (universal but uses more bandwidth), VP9 (saves bandwidth), AV1 (saves even more bandwidth, requires powerful hardware).
  • Containers: MP4 (maximum compatibility), WebM (web optimization).
  • One video upload becomes 15-20 files.
• Solution: DAG (Directed Acyclic Graph):
  • Model the workflow as a DAG.
  • Each processing step is a node, dependencies are edges.
  • The acyclic nature ensures task completion without circular dependencies.
  • Video is split into segments based on keyframes (every 2-10 seconds).
  • Segments are processed independently and in parallel.
  • Video, audio, and metadata take separate paths through the system.
  • Example: Video segments fan out to hundreds of workers for transcoding to different resolutions.
  • Parallel processing reduces processing time from hours to minutes.

Video Streaming

• Challenge: Providing seamless playback under varying network conditions.
• Solution: Adaptive Bitrate Streaming:
  • Video player downloads segments (small chunks of video, a few seconds long) instead of one large file.
  • Player adapts to network bandwidth by switching between different resolutions (e.g., 1080p to 480p).
  • Manifest Files:
    • Primary manifest lists all available formats.
    • Each format has a media manifest with URLs for every segment.
    • Player reads manifests, monitors bandwidth, and fetches appropriate segments.
  • HTTP Range Requests:
    • Player uses HTTP range requests to fetch specific segments (e.g., bytes 1000-2000 of segment 5 of the 720p version).
    • Enables instant seeking.
  • CDN (Content Delivery Network):
    • Segments are stored in CDNs for geographic proximity.
    • Viewers get segments from nearby edge servers, reducing latency.

Further Considerations

• Hot Video Problem: Handling millions of simultaneous requests for viral videos. Requires metadata caching and database hotspot prevention.
• Cost Optimization: Balancing transcoding costs with user experience. Consider transcoding popular formats first or using on-demand transcoding for rare formats. Determine when to migrate to cold storage.
• Pipeline Optimization: Reducing latency by processing segments as they arrive instead of waiting for the complete upload. Pipeline upload and processing for faster availability.
• Geographic CDN Placement: Optimizing CDN placement based on read/write ratios and lazy transcoding strategies.

Synthesis/Conclusion

The design of YouTube's video upload and streaming system relies on three core principles: direct uploads using pre-signed URLs to offload traffic from API servers, DAGs to parallelize the transcoding process, and adaptive bitrate streaming to ensure smooth playback under varying network conditions. These principles are applicable to other large-scale systems dealing with large files, such as file sharing platforms, machine learning pipelines, and live streaming services. Understanding these fundamentals is crucial for designing scalable and efficient systems.