A news feed

1 · Clarifying questions

2 · Capacity math, on a napkin

The 40B writes/day is the headline number. Fan-out-on-write is feasible — but only for non-celebrities. The celebrity column is where the design breaks if you're not careful.

3 · API and data model

Endpoints

POST /v1/posts # create a post
{
 "text": "...",
 "media_ids": ["img-abc", "vid-def"] # already uploaded
}
→ 201 {"post_id":"p_aB3x9Q2", "created_at":"2026-05-09T..."}

GET /v1/feed?cursor=<opaque> # read feed (chronological)
→ 200 {
 "posts": [{ post_id, author_id, text, media, created_at, ... }, ...],
 "next_cursor": "<opaque>"
}

POST /v1/follow # follow another user
{"target_user_id": "u_bob"}
→ 204

DELETE /v1/follow/{target_user_id} # unfollow
→ 204

Schemas

posts -- the source of truth, sharded by post_id
 post_id BIGINT (Snowflake) PRIMARY KEY -- time-ordered ID
 author_id BIGINT NOT NULL
 text TEXT
 media JSONB -- references to media-svc
 created_at TIMESTAMP NOT NULL
 reply_to BIGINT NULL -- for threading
 deleted BOOLEAN NOT NULL DEFAULT FALSE
 INDEX (author_id, created_at) -- "user's posts"

follows -- sharded by follower_id
 follower_id BIGINT
 followee_id BIGINT
 created_at TIMESTAMP
 PRIMARY KEY (follower_id, followee_id)
 INDEX (followee_id, follower_id) -- reverse lookup for fan-out

feeds -- one row per user, chronological list
 user_id BIGINT PRIMARY KEY
 posts LIST<post_id> -- last 1,000; older trimmed
 last_synced TIMESTAMP

celebrity_posts -- the pull-side index for big accounts
 author_id BIGINT
 post_id BIGINT
 created_at TIMESTAMP
 PRIMARY KEY ((author_id), created_at DESC, post_id)
 -- per-author, time-ordered

4 · High-level architecture

Two paths through the system. Write path: post svc → posts KV → fan-out queue → workers → write to each follower's feed. Read path: feed svc → feed cache → merge with celebrity-pull results → optional ranker → return.

5 · The hard part — fan-out strategy

Three approaches. Each has a regime where it wins.

The hybrid algorithm

On POST(author=A, post=p):
 if A.follower_count > CELEBRITY_THRESHOLD (1M):
 INSERT INTO celebrity_posts (author=A, post=p)
 else:
 publish (A, p) to fan-out queue
 workers consume → for each follower F of A:
 LPUSH feeds(F).posts = p (cap at 1000)

On READ_FEED(user=U):
 base = feeds(U).posts -- already-fan-out'd posts
 cels = []
 for each celebrity C that U follows:
 cels += celebrity_posts(C).slice(since=U.last_seen) -- pull
 merged = sort(base + cels) by created_at DESC
 optionally → ranker.rerank(merged)
 return first N

6 · Caching

• Post cache. The post body keyed by post_id in Redis. Hit rate > 99% during traffic peaks. ~24-hour TTL with explicit invalidation on edit/delete.
• Feed cache. Each user's most-recent 200 posts pre-merged and ranked. Refreshed on read if stale (> 60 s old) or on push from fan-out workers. ~80% of feed reads serve from this cache without any database hit.
• Celebrity cache. The last 200 posts per celebrity in a Redis sorted set. Read by every follower's feed-read; high TTL (5 min).
• User-graph cache. "Who does U follow" — small, hot, and invalidated only on follow/unfollow. Often pinned in process-local cache for the top 1% of users.

7 · Ranking, dedupe, freshness

Ranking

Chronological is the default. The "For You" surface plugs in a ranker after the merge — typically a two-stage system:

1. Candidate generation. The merged feed (fan-out + celebrity pull + injected ads + suggested follows) is the candidate pool. ~1,000 candidates.
2. Scoring. A lightweight ML model scores each candidate (engagement probability, freshness decay, source diversity). Run inline at feed-read time.
3. Heavy reranking. Top ~200 go through a heavier model (transformer) before being returned.

Dedupe

A retweet of a post the user has already seen is a dedupe target. Track a per-user "seen post" set (Bloom filter, 1% FP rate, 100 KB per user) — cheap and avoids the worst feed-thrash. Wipe weekly.

Freshness

Time-decay in the ranker: score = engagement / (age_minutes + 120)1.8. This is approximately the Hacker News rank formula. Tune the exponent so the feed isn't dominated by one-hour-old viral posts.

8 · Failure modes & runbook

9 · Cost & SLOs

SLOs

• Feed read availability: 99.99%. Multi-region read; fall back to chronological if ranker fails.
• Feed read P99: 200 ms. Cache-served target; degraded path is < 500 ms.
• Post create availability: 99.95%. Tighter than feed read — losing a post is a product severity.
• Post visibility latency: P99 < 5 s from create to visible in followers' feeds (chronological).

10 · Trade-offs & "what would you change at 10×"

Defending the decisions — interview pushback

The feed question is really six choices in a trench coat. Each draws probing — have a one-sentence answer ready and the rest of the interview goes faster.

1. "Fan-out on write for the common case — why?" Most users don't have 100k followers. For them, fan-out on write is simpler, faster to read, and the write amplification is manageable.
2. "Fan-out on read for celebrities — at what threshold?" 100k to 1M followers is where the math breaks. Pick a number, commit to it, say you'd instrument and tune it. A "celebrity merger" pulls recent posts at feed-load and interleaves them with the pre-computed timeline.
3. "Why cap the pre-computed timeline at ~800 entries?" A Redis sorted set with 800 posts per user is the hot cache. Older posts come from durable storage on demand. LRU eviction handles the rest. Holding everything is unnecessary; holding the hot prefix is enough.
4. "Why cursor pagination, never offset?" Offset breaks when posts are inserted between page loads — the user sees duplicates or skips. Cursor is stable across concurrent writes because it's a monotonic key, not a position count.
5. "Why eventually consistent counters?" Like counts are read by everyone, written by anyone, displayed within milliseconds, and absolutely cannot be exact. Redis HINCRBY for in-flight, periodic sync to durable storage. Nobody can tell 14,823 from 14,824.
6. "Why is ranking separate from feed delivery?" Different SLOs, different failure modes. Feed delivery should not block on an ML model. Ranking is a candidate-set re-ordering step, not part of the retrieval path.

Further reading

• Krikorian (Twitter, 2013) — "Timelines at Scale". The talk that introduced "fan-out on write with celebrity pull" to the wider industry. 30 minutes; required watching.
• Pinterest Engineering — "Smart Feed: building scalable timelines". A cleaner public writeup of the merge-then-rank pipeline.
• LinkedIn Engineering — "FollowFeed: LinkedIn's Feed Made Faster and Smarter". The infra view from a network with very different follow-distribution dynamics.
• Meta Engineering — "Scaling the Pre-Computed Photo Feed Service". The Instagram side. Different content type, same pattern.
• Stack — "How We Designed Our Feed and Reached 100M Users". Useful as a counterpoint at smaller scale.
• Adjacent: URL shortener. Same hot-KV read pattern at smaller payload.
• Adjacent: Replication. The cross-region story.
• Adjacent: Distributed IDs. Snowflake gives time-ordered post IDs that simplify feed merging.