FAANG System Design Interview: Design A Chat System (WhatsApp, Facebook Messenger, Discord, Slack)

Chat System Design at Scale: A Summary

Key Concepts:

• Hybrid Protocol: HTTP for sending messages, WebSocket for receiving.
• Salus Services: Authentication, user profiles, message sending (REST APIs).
• Staple Services: Core chat functionality, chat servers.
• Third-Party Integrations: Push notifications (APNS, FCM).
• Inbox Pattern: Reliable message delivery for offline users.
• Service Discovery: Routing messages between chat servers.
• Fan-Out Pattern: Efficient group message delivery.
• Heartbeats: Tracking online presence.
• Sharding: Distributing database load across multiple servers.

1. Introduction and Core Features:

The video addresses the design of a real-time messaging system capable of handling billions of messages daily, similar to WhatsApp or Facebook Messenger. The focus is on core features:

• One-on-one and group chats (up to 100 participants).
• Guaranteed message delivery.
• Temporary message storage for offline users.
• Online presence indication (green dot).
• Scalability from thousands to millions of users.

2. Client-Server Communication Protocols:

• HTTP: Suitable for sending messages (client-initiated POST requests).
• Polling: Inefficient for receiving messages due to wasted server resources.
• Long Polling: Improves upon polling but still not optimal.
• WebSocket: Persistent, bidirectional communication channel ideal for real-time updates (server push). However, scaling WebSocket connections can be challenging.

The chosen approach is a hybrid model: HTTP for sending messages (scalability) and WebSocket for receiving messages (real-time updates).

3. Overall System Architecture:

The system is divided into three layers:

• Salus Services:
  • Handle authentication, user profiles, and message sending.
  • Utilize REST APIs.
  • Employ load balancers for horizontal scalability.
• Staple Services:
  • Form the core chat functionality.
  • Consist of chat servers that maintain persistent WebSocket connections with clients.
  • Server capacity (tens or hundreds of thousands of concurrent connections) dictates the number of servers needed.
• Third-Party Integrations:
  • Handle push notifications for offline users (e.g., Apple's APNS, Google's FCM).
  • Leverage existing, scalable solutions for push notifications.

4. Handling Offline Scenarios: The Inbox Pattern:

The inbox pattern ensures message delivery even when users are offline:

• Each user has a personal queue (inbox) for storing undelivered messages.
• When a message is sent:
  • If the recipient is online, the message is delivered via WebSocket, and an acknowledgment (ACK) is sent back. No storage is needed.
  • If the recipient is offline or delivery fails, the message is stored in the recipient's inbox.
• Acknowledgment Mechanism:
  • When a device receives a message from the inbox, it sends an ACK.
  • The server removes the message from the inbox only after receiving the ACK.
  • If no ACK is received, the server retries delivery later.
• When a user comes back online, their chat server retrieves waiting messages from their inbox and delivers them in order.

5. Message Routing Between Chat Servers: Service Discovery and Direct Communication:

To handle scenarios where users are connected to different chat servers, the system uses:

• Service Discovery: A user presence service tracks which chat server each user is connected to. This service acts as a directory.
• Direct Server Communication (RPC):
  • When Alice sends a message to Bob, Alice's server checks Bob's online status via the user presence service.
  • If Bob is online, Alice's server makes a direct RPC call to Bob's server.
  • Bob's server pushes the message to Bob's device via WebSocket.
  • If Bob is offline, the message goes to his inbox, and a push notification is triggered.
• This approach minimizes latency and scales well. Discord is mentioned as an example of a platform using this pattern.

6. Group Chat: The Fan-Out Pattern:

For group chats (up to 100 members), the system uses a fan-out pattern:

• When a user sends a group message, the server checks which members are online.
• The server makes RPC calls to deliver the message immediately to the online members' devices.
• Offline members receive the message in their inboxes.
• This approach scales with group size.

7. Online Presence: Heartbeats:

Online presence (the green dot) is tracked using heartbeats:

• Clients send WebSocket ping frames to their chat server every 30 seconds.
• The server tracks the timestamp of the last ping received for each user.
• If the server receives no ping for 60 seconds, it marks the user as offline.
• This approach accounts for brief disconnections.

8. Scaling Challenges and Solutions:

As the user base grows, the following challenges arise:

• Connection Limits: Chat servers become the first bottleneck. Solution: Add more chat servers.
• Database Limits: Inboxes fill up and empty constantly. Solution: Shard users across multiple database servers based on user IDs.
• Global Reach: Deploy chat servers in multiple regions. Users connect to their nearest region for low latency, but cross-region message routing adds complexity.

9. Additional Considerations (Briefly Mentioned):

• Message Ordering: Message IDs, timestamps, and vector clocks.
• Security and Encryption: End-to-end encryption, key exchange mechanisms.
• Media Handling: Compression, CDNs, progressive loading.
• Read Receipts and Typing Indicators: Careful design to avoid overwhelming the system.
• Rate Limiting and Abuse Prevention: Balancing spam prevention with user experience.

10. Conclusion:

The video provides a high-level overview of the core architectural components and design considerations for building a scalable real-time chat system. It emphasizes the importance of choosing appropriate communication protocols, implementing reliable message delivery mechanisms, and addressing scaling challenges as the user base grows. The hybrid HTTP/WebSocket approach, the inbox pattern, service discovery, and the fan-out pattern are key elements in achieving a robust and scalable solution.