A notification system

1 · Clarifying questions

2 · Capacity math, on a napkin

The cost math says SMS dominates. Most of the design's cost discipline goes into "do not send an SMS the user could have received as a push" — channel selection is half the architecture.

3 · API and data model

Producer API (called by every other service)

POST /v1/notifications # send a notification
{
 "user_id": "u_abc",
 "category": "payment_confirmation", # required; routes to template + prefs
 "params": {"amount": "$42.00", "merchant": "..."},
 "idempotency_key": "pmt_xyz_2026-05-09", # client-provided; dedupe key
 "channels": ["push", "email"], # optional override
 "priority": "transactional" # transactional | bulk
}
→ 202 {"notification_id": "n_aB3x9Q2"}

POST /v1/notifications/bulk # marketing / digest fan-out
{
 "audience_query": "active_users_who_purchased_last_7d",
 "category": "weekly_digest",
 "template_id": "digest_v3",
 "schedule_at": "2026-05-10T09:00:00Z"
}
→ 202 {"job_id": "job_..."}

User-facing endpoints (preferences)

GET /v1/users/:id/preferences # categories × channels
PUT /v1/users/:id/preferences/:category # opt in/out
GET /v1/users/:id/devices # registered tokens
POST /v1/users/:id/devices # register a new APNs/FCM token
DELETE /v1/users/:id/devices/:device_id

Storage

notifications -- 90-day retention
 notification_id BIGINT PRIMARY KEY
 user_id BIGINT
 category VARCHAR(64)
 channels_planned JSONB -- e.g., ["push","email"]
 channels_sent JSONB -- after delivery attempts
 idempotency_key VARCHAR(64) UNIQUE -- producer-provided
 payload JSONB
 status VARCHAR(16) -- queued|sent|failed|suppressed
 created_at TIMESTAMP
 INDEX (user_id, created_at)
 INDEX (idempotency_key)

user_preferences
 user_id BIGINT
 category VARCHAR(64)
 channel VARCHAR(16)
 enabled BOOLEAN
 PRIMARY KEY ((user_id), category, channel)

devices -- push tokens
 device_id UUID PRIMARY KEY
 user_id BIGINT
 platform VARCHAR(16) -- ios|android|web
 token TEXT
 last_seen TIMESTAMP
 status VARCHAR(16) -- active|inactive|revoked

suppressions -- regulatory bounces
 channel VARCHAR(16)
 identifier VARCHAR(255) -- email | phone
 reason VARCHAR(64) -- hard_bounce | unsubscribed | complained
 added_at TIMESTAMP
 PRIMARY KEY ((channel), identifier)

4 · High-level architecture

Producers post to the API. The API does idempotency check, fetches the user's preferences and the template version, then publishes one Kafka event per (notification, channel) tuple. Dispatcher consumes, calls render-svc, hands off to the right channel adapter. Adapters own per-provider retry + rate-limiting + cost tracking.

5 · The hard part — channel selection & deduplication

Two flavours of "don't send this twice" land here. They look similar and are not.

Idempotency at the API

Every producer call carries an idempotency_key. The notifications table has a unique index on it. A second call with the same key returns the original notification — no second send. This catches retry storms from upstream services.

Cross-event dedupe (semantic)

A user gets 5 likes in 30 seconds. We don't want to send 5 push notifications. The pattern is "coalescing" — the producer sends each event, the dispatcher groups recent same-category events for a user, and the adapter sends one consolidated notification at the end of a short window (10–30 s for push, 5 minutes for email).

# Coalescing window per (user, category)
key: coalesce:u_abc:likes
zset: { event_id_1: ts_1, event_id_2: ts_2, ... }
TTL: 60s

# Dispatcher logic
on receive event(user, category, ...):
 ZADD coalesce:user:category event_id ts
 if ZCARD = 1:
 schedule send_now_or_after(window)
 else:
 no-op (already scheduled)

on timer fires:
 events = ZRANGE coalesce:user:category 0 -1
 ZDEL all
 template = pick("you have 5 new likes" if events > 1 else "alice liked your post")
 send

Channel selection

The decision tree at send time:

1. Look up user_preferences[user, category]. Filter channels to those the user has opted in to.
2. For each remaining channel, check suppressions — is this email/phone hard-bounced or unsubscribed? Drop.
3. For transactional (priority), send to all remaining channels.
4. For bulk, prefer cheap channels — push first, email if no push, SMS only as a deliberate product choice.
5. If user is currently online (active push session in the last 5 min), suppress email for this notification — they'll see the push.
6. Mark the chosen channels in channels_planned; emit one Kafka event per channel.

6 · Retries, dead letters, and the bounce loop

Each adapter implements retry independently — the rates, error codes, and bounce semantics are all channel-specific.

7 · Templates, A/B, and rendering

• Templates as config. Versioned in Git, deployed to a key-value store. Each notification carries a template_id and the params; the renderer combines them.
• Multi-language. Locale-keyed template variants. Locale is on the user record; fallback to English.
• A/B. Two template variants under one logical id; dispatcher hashes (user_id, template_id) into a bucket. Outcomes (open, click, conversion) tracked back via a separate analytics pipeline.
• Rendering at send-time. Don't pre-render at queue time — the user's locale or preference may have changed. Renderer is stateless; it fans out per request.
• Sanitisation. Strip HTML from any user-generated content interpolated into templates. The Comment-poster's name is the most common XSS vector for email bodies.

8 · Failure modes & runbook

9 · Cost & SLOs

SLOs

• Transactional P99: 5 s end-to-end. API → Kafka → dispatcher → adapter → provider → user device. Provider is the dominant cost.
• Bulk P99: 30 minutes. Coalescing + scheduled delivery; tighter for the digest jobs.
• Delivery rate: ≥ 95% per channel. Lower than 95% triggers reputation review.
• Bounce-to-suppression latency: P99 ≤ 60 s. Bounces must propagate before the next dispatch fires.
• Unsubscribe-to-stop: P99 ≤ 5 minutes. Regulatory; not an SLO we can negotiate.

10 · Trade-offs & "what would you change at 10×"

Further reading

• Slack Engineering — "Reducing Slack's memory footprint". Tangential, but the section on push-fan-out is widely cited.
• Pinterest Engineering — "Building Pinterest's notification platform". The clearest public writeup of a multi-channel pipeline at scale; the abstraction layers map almost 1:1 onto this design.
• Lyft Engineering — "Asynchronous notifications at Lyft". The Kafka-fan-out architecture in production; useful operational details.
• Twilio — "Programmable Messaging API". Read the SMS API docs; the failure modes and cost model are foundational.
• Amazon SES — "Best practices for sending email". The bounce/complaint handling section is required reading for any email-sending system.
• FTC — "CAN-SPAM Act: A Compliance Guide for Business". Short. The mistake-cost is real — read it.
• Adjacent: Chat. Push-when-offline plumbing overlaps significantly with chat's offline path.
• Adjacent: Message queues. The Kafka layer.
• Adjacent: Idempotency. The producer-side pattern.