Object storage

1 · Clarifying questions

Five minutes. Durability targets and object size range matter most — they drive the erasure scheme, the chunk size, and the metadata footprint.

2 · Capacity math, on a napkin

S3-scale public numbers. They sound enormous but they decompose into well-understood subsystems — a sharded KV for metadata, a fleet of chunk servers for bytes.

100 TB of metadata sounds large but it's tractable in a sharded KV. The bytes are the expensive line — storage cost dominates the bill, not request cost.

3 · API and data model

S3-compatible on the outside. Inside, the metadata plane and the data plane are separate services with separate scaling stories.

Endpoints

PUT  /:bucket/:key                  # single-shot upload (≤ 5 GB)
     body: object bytes
     headers: Content-Length, Content-Type, x-amz-meta-*
  → 200 {"etag":"...", "version":"..."}

POST /:bucket/:key?uploads          # initiate multipart
  → 200 {"upload_id":"..."}

PUT  /:bucket/:key?partNumber=N&uploadId=...   # upload a part (5 MB – 5 GB)
  → 200 {"etag":"part-etag"}

POST /:bucket/:key?uploadId=...     # complete (list of part etags)
  → 200 {"etag":"final", "version":"..."}

GET  /:bucket/:key                  # download (supports Range)
  → 200 body

DELETE /:bucket/:key                # tombstone (or new delete-marker version)
  → 204

GET  /:bucket?prefix=&marker=&max-keys=
  → 200 list of keys, strongly consistent

Schema (metadata plane, sharded KV)

object_metadata
  bucket        STRING
  key           STRING              -- arbitrary UTF-8, up to 1024 bytes
  version       STRING              -- monotonic per (bucket, key)
  size          BIGINT
  etag          STRING              -- MD5 (single-shot) or composite (multipart)
  content_type  STRING
  chunk_map     LIST<chunk_ref>     -- ordered list of chunk IDs + offsets
  acl           STRING              -- inline if small, else reference
  created_at    TIMESTAMP
  is_delete_marker BOOLEAN
  storage_class STRING              -- HOT / WARM / COLD

  PRIMARY KEY ((bucket, key), version DESC)
  shard by hash(bucket, key)

chunk_locations
  chunk_id      STRING (UUID)
  shards        LIST<shard_ref>     -- 9 entries for 6+3, with rack + AZ
  size          INT
  PRIMARY KEY (chunk_id)
  shard by hash(chunk_id)

Two tables, two sharding keys. object_metadata answers "what chunks make up this object". chunk_locations answers "where on disk does this chunk live". The split lets the placement service rebalance chunks without touching object metadata.

4 · High-level architecture

Two planes that share an API gateway. The metadata plane is small, hot, and sharded. The data plane is large, cold(er), and parallel.

Data flows separately from metadata. The gateway authenticates; the router hashes; the metadata service hands back chunk locations; the client (or the gateway, for small objects) streams bytes directly to chunk servers. The repair scanner runs continuously in the background, rebuilding shards from parity wherever it finds gaps.

5 · Durability — erasure coding and placement

How eleven nines is actually paid for. Triple replication is the simple answer; erasure coding is the answer at scale.

The recommendation: Reed-Solomon 6+3 across 3 AZs. Each chunk becomes 9 shards — 6 data, 3 parity — placed 3 per AZ. The system tolerates the loss of any 3 shards simultaneously, which covers a disk, a rack, or an entire AZ. Storage overhead is 50% versus 200% for replication; for a 100 PB store that is a 150 PB bill rather than 300 PB.

6 · Multipart upload

A 5 TB object cannot be a single HTTP request — too much sits in flight, retries are catastrophic, and no client survives a mid-flight TCP drop on a 12-hour upload. Multipart breaks the upload into independently-uploaded parts that the server stitches together at the end.

• Initiate. Client POSTs to get an upload_id. Server records "in-progress upload" in the metadata plane.
• Upload parts in parallel. Each part is 5 MB – 5 GB. Parts can upload in any order, can be retried independently, and each returns an etag. Up to 10,000 parts per upload.
• Complete. Client sends the list of part numbers + etags. Server validates each part exists, stitches the chunk_map, and writes the final metadata in one commit. The composite etag is the MD5 of the concatenated part etags (which is why S3 etags for multipart objects aren't the file's MD5).
• Abort. Failed uploads can be explicitly aborted, or the GC scanner sweeps parts older than ~7 days. Without GC, abandoned multipart uploads accumulate forever as billed dark storage.

7 · Failure modes & runbook

The operational story. Most of these are continuous background processes rather than page-oncall events — the system is designed to absorb constant low-level failure.

8 · Cost, operability, and trade-offs

Storage is the cost line that dominates everything else. Request and bandwidth costs matter but are usually secondary; the spreadsheet for an object store is mostly $/GB-month × total bytes.

Trade-offs and what's next

• Strong vs eventual consistency. S3 went strong in 2020 by changing how listings are built — metadata shards serve lists directly. The cost is more load on the metadata plane; the payoff is that read-after-write applies to listings too.
• Cross-region replication. Active-passive (async, RPO ≈ 15 min) is straightforward — a background pipe replicates new versions to a peer region. RPO depends on the pipe's throughput vs the write rate.
• Multi-region active-active writes. Much harder than read replicas. Concurrent writes to the same key need conflict resolution — last-writer-wins on timestamp is the common compromise, with the honest caveat that clock skew loses writes.
• Conditional writes. S3 added If-Match / If-None-Match in 2024 for optimistic concurrency. Lets clients do compare-and-swap without an external lock service. Closes a long-standing gap for databases-on-object-storage.
• Erasure scheme as a tier. Hot data on 6+3 (low repair latency); archival on 10+4 (better overhead, slower repair). Picking one scheme for all tiers leaves money on the table.

Defending the decisions — interview pushback

Six choices an interviewer will probe. Have a one-sentence answer ready for each — the depth of these defenses, more than the diagram itself, is what separates a "passing" from a "strong-hire" signal.

1. "Why separate metadata and data planes?" Different SLOs, different access patterns, different scaling stories. Metadata is small, indexed, needs atomic updates. Data is large, sequential, immutable once written. Mixing them means you compromise both.
2. "Why erasure coding instead of 3× replication?" At petabyte scale, 3× means 200% overhead. A 6+3 erasure code costs ~50%. The math is straightforward once seen; at exabyte scale it's not optional.
3. "Why pre-signed URLs?" Application servers should never proxy bytes for large objects — it pins connections, double-pays for bandwidth, adds latency. The server mints a signed URL; the client uploads directly to storage.
4. "Why two-phase write — data first, metadata second?" Data first is the recoverable direction. Metadata first plus a data failure leaves a tombstone pointing nowhere. Data first plus a metadata failure leaves orphan blobs that GC reclaims. Always write the hard-to-recover thing last.
5. "Why a continuous repair scanner?" Disk failures happen constantly at fleet scale. The scanner reconstitutes missing shards from surviving parity. If repair rate falls below failure rate, durability silently degrades — and you don't notice until it's too late.
6. "Why lifecycle policies as first-class?" Most data goes cold after 30 days. Automatic tiering to cheaper storage is a 5–10× cost reduction per byte — a feature you build in, not an afterthought.

Further reading

• AWS — "Diving Deep on S3 Consistency" (Andy Warfield, re:Invent 2021). The public account of how S3 went from eventual to strong listings without an outage. The architectural detail is worth the talk's full hour.
• Facebook — "f4: Facebook's Warm BLOB Storage System" (OSDI 2014). Reed-Solomon erasure coding for a photo-store workload. Pairs nicely with the older Haystack paper for hot/warm/cold tiering.
• Google — "Colossus: Successor to GFS". The metadata/data split, the chunk server model, and how erasure coding replaced 3× replication.
• Backblaze — "Reed-Solomon erasure coding from scratch". A walkthrough of the math with working Java. The most accessible introduction to RS that exists.
• James Hamilton — "On Designing and Deploying Internet-Scale Services". Not object-storage-specific, but the operational guidance ("expect failure", "design for many small components") shaped how S3 was built.
• Adjacent: File storage design handbook. The block / file / object spectrum and when each fits.
• Adjacent: How CDNs work. The layer that fronts an object store for public-read workloads.