Design Instagram

1 · Clarifying questions

2 · Capacity math, on a napkin

The two headline numbers: 1.4 Tbps ingest at peak (so uploads go straight to object storage, not through your app servers) and the CDN doing ~95% of egress (so the origin sees a tiny fraction of the read traffic).

3 · API and data model

Upload flow

POST /v1/uploads/init                  # client requests upload
{ "filename": "img.jpg", "size_bytes": 4194304, "content_type": "image/jpeg" }
→ 200 {
  "upload_id": "u_8aB3x9Q2",
  "presigned_url": "https://...s3...?...",  # client uploads directly to object storage
  "parts": 4,                                  # multipart for files >5 MB
  "expires_at": "..."
}

POST /v1/uploads/:id/complete          # after S3 multipart complete
→ 201 { "photo_id": "p_8aB3x9Q2", "status": "processing" }

GET  /v1/photos/:id                    # metadata + URLs to processed variants
→ 200 {
  "id": "p_8aB3x9Q2",
  "owner": "u_abc",
  "caption": "...",
  "variants": {
    "150":  "https://cdn.../p_8aB3x9Q2/150.jpg",
    "320":  "...",
    "640":  "...",
    "1080": "..."
  },
  "status": "ready"
}

GET  /v1/users/:id/feed                # follow-graph feed

Storage

photos                           -- Cassandra; partitioned by owner_id
  photo_id      BIGINT PRIMARY KEY  -- Snowflake; time-ordered
  owner_id      BIGINT
  caption       TEXT
  raw_key       TEXT                 -- S3 key of original
  variant_keys  MAP<TEXT, TEXT>      -- "150" → s3 key, etc.
  status        VARCHAR(16)          -- uploading | processing | ready | failed
  created_at    TIMESTAMP
  INDEX (owner_id, created_at)

photo_metadata                   -- exif, dims, hashes
  photo_id      BIGINT PRIMARY KEY
  width, height INT
  exif          JSONB
  phash         BIGINT               -- perceptual hash for dedupe
  sha256        VARCHAR(64)

likes                            -- one row per (user, photo)
  user_id, photo_id BIGINT
  liked_at TIMESTAMP
  PRIMARY KEY ((photo_id), user_id)

follow_graph                     -- as Twitter

feed                             -- precomputed home feed
  user:{id}:feed → ZSET<photo_id, score>
                   (kept to ~500 ids)

object_store (S3-shape)
  /raw/{photo_id}                -- original; cold tier after 30 days
  /variants/{photo_id}/{size}.jpg -- served via CDN

4 · High-level architecture

Client uploads to S3 directly via a presigned URL — the API server never holds the bytes. On complete, the API enqueues a processing job. Workers fetch the raw image, generate variants, write them back to a different bucket, update metadata, then run moderation and emit a fan-out event. Reads serve from the CDN; the origin only fires on cache miss.

5 · The hard part — the upload + processing pipeline

Why direct-to-S3?

At 1.4 Tbps ingest, you cannot push bytes through your app servers. Two reasons: bandwidth cost (you're paying for ingress twice — client → app, app → S3), and CPU cost (your app servers become I/O-bound babysitters). Presigned URLs let the client write directly to S3. The API only deals in small JSON messages — init request, complete callback, status checks.

Multipart upload

Files > 5 MB get split into 5 MB parts. Each part uploaded in parallel; one failed part is retried, not the whole upload. Critical on mobile networks where a 10 MB upload over 4G has a real failure rate. On complete, S3 stitches the parts into one object and the API gets the callback.

Processing pipeline

# Worker pulls from Kafka topic 'photos_uploaded'
def process(photo_id):
  raw = s3.get(f"raw/{photo_id}")
  exif = extract_exif(raw)
  phash = perceptual_hash(raw)               # for dedupe + moderation lookup
  sha = sha256(raw)

  if dedupe.is_known_bad(phash):              # NSFW, CSAM, malware
    mark_status(photo_id, "rejected")
    return

  variants = {}
  for size in [150, 320, 640, 1080]:
    img = resize_and_compress(raw, size, quality=85, format="jpeg")
    key = f"variants/{photo_id}/{size}.jpg"
    s3.put(key, img, cache_control="public,max-age=31536000,immutable")
    variants[size] = key

  cassandra.update(photo_id, {
    "variant_keys": variants,
    "phash": phash,
    "sha256": sha,
    "exif": exif,
    "status": "ready"
  })

  kafka.send("photo_ready", {photo_id, owner_id})  # triggers fan-out

Each worker handles ~10 photos/sec (CPU-bound on resize). At 44K peak uploads/sec, we need ~5,000 worker pods. Auto-scale by Kafka lag.

The CDN cache-control story

Variants are written with Cache-Control: public, max-age=31536000, immutable. The URL contains the photo_id, which is content-addressed by the variant — so the object at that URL never changes. The CDN can hold it forever. This is what gets the cache-hit ratio to 95%+.

Edits are a new photo_id (or at minimum a new URL with a version suffix). Deletes are handled by tombstoning the metadata and letting the CDN entries expire naturally; URLs become 404 once the object is gone from S3.

6 · Storage tiering

Variants are kept hot — they're the read path. Originals can move to colder tiers because we rarely re-process old photos. The lifecycle policy moves objects automatically based on age.

7 · Failure modes & runbook

8 · Cost & SLOs

SLOs

• Upload "feels instant". First-byte to client < 200 ms (the presigned URL request). Bytes go straight to S3 — client measures upload speed.
• Photo ready P99: 30 s. From upload complete to visible in profile.
• Feed load P99: 300 ms. Feed metadata read; variant URLs included; CDN handles the bytes.
• CDN hit ratio ≥ 95%. Anything lower and the origin egress bill explodes.
• Durability: 11 nines. S3-class. Cross-region replication for hot data; cross-region async for cold.

9 · Trade-offs & "what would you change at 10×"

Further reading

• Instagram Engineering blog — multiple posts on Cassandra, sharding, the photo pipeline. The clearest first-party source on this design.
• "What Powers Instagram: Hundreds of Instances, Dozens of Technologies". An early but still-instructive overview of the stack.
• "Sharding & IDs at Instagram". Their Snowflake-style ID scheme; relevant to the photo_id field used throughout.
• Adjacent: Object storage. The S3-shape underneath.
• Adjacent: CDN. The pattern that makes the read path affordable.
• Adjacent: News feed. The feed delivery; same pattern, different payload.
• Adjacent: Napkin math. The Instagram preset is in the live worksheet.