Chat & direct messaging

1 · Clarifying questions

2 · Capacity math, on a napkin

The 10M concurrent connections is the architecture-defining number. Everything else flows from it: 200 gateway pods, ~$3M/month in compute alone, the per-connection budget that determines what state a pod can hold per user.

3 · API and data model

WebSocket frames (the hot path)

# After WS handshake + auth, every frame is a small JSON or Protobuf
# Use Protobuf for production. JSON shown here for readability.

# Send
→ { "op": "send", "msg_id": "<client-uuid>", "conv": "c_abc",
 "body": "...", "media": [...] }
# Server ack
← { "op": "ack", "msg_id": "<client-uuid>", "server_id": "m_x9Q2", "ts": 1746... }

# Inbound delivery
← { "op": "msg", "server_id": "m_x9Q2", "conv": "c_abc",
 "from": "u_alice", "body": "...", "ts": 1746... }
# Client ack (delivered)
→ { "op": "delivered", "server_id": "m_x9Q2" }
# Client read
→ { "op": "read", "server_id": "m_x9Q2" }

# Presence / typing — best effort
→ { "op": "typing", "conv": "c_abc" }
← { "op": "presence", "user": "u_bob", "state": "online" }

REST endpoints (cold-path supporting)

POST /v1/conversations # create DM or group
GET /v1/conversations # list user's conversations (recent)
GET /v1/conversations/:id/messages # paginated history (cursor-based)
POST /v1/conversations/:id/members # group: add members
DELETE /v1/conversations/:id/members/:uid

Storage

messages -- sharded by conversation_id
 conversation_id BIGINT
 message_id BIGINT (Snowflake) -- time-ordered
 sender_id BIGINT
 body BYTES -- Protobuf or ciphertext
 media_refs JSONB
 client_msg_id UUID -- dedupe key
 PRIMARY KEY ((conversation_id), message_id DESC)
 -- Cassandra-style: partition by conv, cluster by time DESC
 -- gives O(1) "newest N messages in conversation"

conversations
 conversation_id BIGINT PRIMARY KEY
 type VARCHAR(8) -- dm | group_small | group_large
 created_at TIMESTAMP
 members SET<BIGINT> -- inline if ≤ 100
 members_count INTEGER -- denormalized
 last_message_id BIGINT -- for "recent conversations"
 meta JSONB -- name, icon (groups)

membership -- reverse index
 user_id BIGINT
 conversation_id BIGINT
 joined_at TIMESTAMP
 last_read_id BIGINT
 muted BOOLEAN
 PRIMARY KEY ((user_id), conversation_id)

connection_registry -- in Redis, NOT durable
 user_id → set<gateway_pod_id, ws_connection_id, last_seen>

4 · High-level architecture

Three layers below the WebSocket gateway. Connection registry (who's connected where), message router (look up recipient → forward to their gateway), and the async lane (Kafka → push to APNs/FCM, write to messages KV). Group fan-out runs through the message bus rather than the connection registry.

5 · The hard part — routing a message to the recipient

Sender and receiver are usually on different gateway pods. The message-router has to find the right pod for the recipient and forward the frame.

6 · Connection lifecycle

Persistent WebSockets are a different operational beast than HTTP. The lifecycle determines what gets paged and when.

• Sticky sessions at the L4 LB. WebSocket reconnects should land back on a healthy pod, but not necessarily the same one. The connection-registry is the source of truth.
• Heartbeat and idle timeout. Client pings every 30 s; server tears down at 90 s without a ping. Mobile flake-rate is the dominant cause of "presence flap" — model it explicitly.
• Graceful drain. Gateway pod restart: stop accepting new connections, wait 60 s, then close existing connections with a 1012 (Service Restart). Clients reconnect to a new pod within seconds.
• Reconnect with backfill. On reconnect, client sends its last_message_id; server replays anything newer from Kafka or messages table. Bounded — a 24-hour gap means the client gets the most-recent N and a "load older" pagination link.
• Multi-device. One user can have 5+ active connections (phone, tablet, web). The connection registry is a set, not a single value. Every device gets the message.

7 · Group fan-out

Three regimes:

8 · Delivery semantics & dedupe

At-least-once with idempotent dedupe at the client. Every message has a client_msg_id (UUID) generated at send-time; the storage layer treats that as a unique key, so a retry produces no second insert.

1. Sender resends if it doesn't see a server ack within 3 s. Same client_msg_id; the messages table treats it as a duplicate write (UPSERT-style).
2. Recipient deduplicates on server_id at the client layer. The client app's local DB is the source of truth for "have I shown this".
3. Read receipts & presence are best-effort. No durability; a lost typing-event isn't worth a retry.
4. Order preserved per conversation. Kafka partition-per-conversation gives this for free.
5. Cross-device sync. Each device holds its own last_message_id per conversation; on reconnect, gateway replays from that mark.

9 · Failure modes & runbook

10 · Cost & SLOs

SLOs

• Send-to-deliver P99: 200 ms when both online. Edge → gateway → Kafka → router → recipient gateway → recipient WS.
• Send-to-push P99: 5 s offline. APNs/FCM is the dominant variable.
• Connection availability: 99.99%. Reconnect-within-10 s after pod failure considered "connected".
• Message durability: 99.999%. Cassandra RF=3 + Kafka RF=3 inside the partition gives this.

11 · Trade-offs & "what would you change at 10×"

12 · Defending the decisions — interview pushback

Most senior interviewers don't ask "what would you build" — they push back on each choice and watch how cleanly you defend it. These are the five exchanges that recur for chat designs; have a one-sentence answer ready for each.

1. "Why WebSocket and not HTTP long-polling?" Bidirectional, low overhead, no repeated headers on every poll cycle. Long polling is a firewall-fallback, not a first choice for a system designed from scratch.
2. "Why Cassandra and not Postgres for messages?" The write rate (580K/s) and the access pattern (latest N per conversation) are exactly what Cassandra's clustering keys optimise for. Postgres would need a write-optimised partition strategy you'd have to build yourself.
3. "Why two Cassandra tables (messages-by-conv, messages-by-user)?" Conflicting read patterns. History view reads by conversation_id; offline sync and push read by user_id. Two tables, same data, different partition key — both patterns serve without a scatter-gather.
4. "Why Kafka in the middle and not a direct gRPC fan-out?" Durability (sender ack means it's in Kafka, not a gateway's RAM) and ordering (per-conversation partition is in-order for free). Skip Kafka and you rebuild both properties from scratch.
5. "What about offline sync — how do you avoid replaying a million messages?" Each device tracks last_message_id per conversation. On reconnect, the gateway replays from that mark only, with a hard cap (most-recent 200, then paginate). Design this on day one; retrofitting it is painful.

Further reading

• Slack Engineering — "Real Time Messaging". Series of posts on Slack's WebSocket gateway architecture. The "flannel" service is the routing layer described above.
• Discord Engineering — "How Discord Stores Trillions of Messages". Cassandra-shaped storage at scale; the per-conversation partition pattern is the core lesson.
• Facebook Engineering — "Building Mobile-First Infrastructure for Messenger". The MQTT-based predecessor; useful as a counter-design.
• WhatsApp Engineering — "1 Million is Just the Beginning". Erlang and the BEAM VM as a connection multiplexer. Different stack, same problem.
• Signal — "Sealed Sender". The end-to-end encryption design that pushed the state of the art.
• Adjacent: TLS. The transport that secures every WebSocket frame.
• Adjacent: WebSockets. The protocol underneath the gateway.
• Adjacent: Message queues. Why Kafka is the right shape for the durable middle.