Design a messaging system with WebSocket connections, message delivery guarantees, and end-to-end encryption.
Published April 23, 2025
Functional: Send/receive messages, Group chats (max 256), Online presence, Message delivery receipts (sent/delivered/read) Scale: 2B users, 100B messages/day, 99.99% availability, message delivered within 1 second
Real-time messaging: use WebSocket connections (persistent bidirectional TCP)
Client A ←─────── WebSocket ──────→ Chat Server A
Client B ←─────── WebSocket ──────→ Chat Server B
A → B message flow:
A → WS → Chat Server A → Message Queue → Chat Server B → WS → B
1. Alice sends "Hello" to Bob
2. Alice's app → WebSocket → Chat Server
3. Chat Server saves message to DB (status: SENT)
4. Chat Server acknowledges: ✓ (sent)
5. If Bob is online:
Chat Server B pushes message via Bob's WebSocket
Bob's app ACKs → status: DELIVERED (✓✓)
Bob reads → status: READ (✓✓ blue)
6. If Bob is offline:
Message stored in DB
When Bob reconnects, server pushes pending messages
Or: push notification via APNs/FCM
-- Messages (Cassandra — high write, time-series)
CREATE TABLE messages (
conversation_id UUID,
message_id TIMEUUID, -- time-sortable
sender_id UUID,
content TEXT,
media_url TEXT,
status TINYINT, -- 1=sent, 2=delivered, 3=read
PRIMARY KEY ((conversation_id), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);
-- Users / Connections
users: user_id, name, phone, avatar_url
connections: user_id → chat_server_id (stored in Redis)
When Bob connects:
Redis: SET conn:bob_id → chat_server_2
Redis: SET presence:bob_id → "online" EX 300
When Alice sends to Bob:
Lookup: GET conn:bob_id → chat_server_2
Route message to chat_server_2 via internal service mesh
chat_server_2 pushes to Bob's WebSocket
Group message (256 members):
→ Fan-out: write message to all 256 members' delivery queues
→ For active members: push via WebSocket
→ For offline members: queue until reconnect
WhatsApp uses Signal Protocol:
1. Each device generates public/private key pair
2. Public key stored on WhatsApp servers
3. Messages encrypted with recipient's public key
4. Only recipient can decrypt (server never sees plaintext)