Pastebin

1 · Clarifying questions

2 · Capacity math, on a napkin

The takeaway: this is a storage problem, not a compute problem. Object storage (S3) at $0.023/GB-month gets us 1.8 PB for ~$40K/month — and most of that flows out via the CDN, which is the next biggest cost line.

3 · API and data model

Endpoints

POST /v1/pastes # create
{
 "content": "...", # required, ≤ 2 MB
 "language": "python", # optional, hint for client-side highlighter
 "privacy": "unlisted", # public | unlisted | private
 "expires": "1d" # 10m | 1h | 1d | 1m | 1y | never
}
→ 201 {"id":"aB3x9Q2", "url":"https://pb.sh/aB3x9Q2", "expires_at":"2026-05-10T..."}

GET /:id # read (the hot path)
→ 200 text/plain + the paste content
 Cache-Control: public, max-age=300, s-maxage=86400

GET /:id/raw # read raw, for clients/tools
→ 200 text/plain

POST /:id/report # abuse / takedown
→ 202

DELETE /:id # owner only
→ 204

Schema

Two tables, plus the blob in object storage. The Postgres row is intentionally tiny — the paste body lives in S3 with the row pointing at the object key.

pastes -- metadata only, ~300 B / row
 id VARCHAR(8) PRIMARY KEY -- base62 ID
 blob_key VARCHAR(64) NOT NULL -- s3://pastes/<sha256-prefix>/<sha256>
 size_bytes INTEGER NOT NULL
 language VARCHAR(32) NULL
 privacy VARCHAR(16) NOT NULL -- public | unlisted | private
 owner_id BIGINT NULL
 content_hash CHAR(64) NOT NULL -- sha256, used for dedupe
 created_at TIMESTAMP NOT NULL
 expires_at TIMESTAMP NULL
 deleted_at TIMESTAMP NULL -- soft delete; reaper hard-deletes after grace
 flagged BOOLEAN NOT NULL DEFAULT FALSE -- abuse review

 INDEX (expires_at) WHERE expires_at IS NOT NULL -- expiry sweep
 INDEX (content_hash) -- dedupe lookup
 INDEX (owner_id, created_at) -- "my pastes"
 INDEX (created_at) WHERE privacy = 'public' -- public listing

paste_blobs -- S3 bucket, content-addressed
 Bucket layout: pastes/<first-2-hex>/<sha256>
 Storage class: Standard for hot, Infrequent Access after 30d, Glacier after 1y
 Server-side encryption: SSE-S3 (free) or SSE-KMS (compliance)
 Lifecycle policy: tier-down on age, hard-delete on expiry

Content-addressed storage by SHA-256 means duplicate pastes — and there are many — share a blob. The pastes table holds N rows pointing to one S3 object. ~30% storage savings in practice.

4 · High-level architecture

The shape is read-side-heavy. The edge does most of the work; the origin handles creates and the long tail of cache misses.

The hot path on read is: edge → read svc → Redis (for the metadata) → S3 (for the blob). On a CDN hit, none of this fires. The create path is: create svc → scan svc (async, can fail-open or fail-closed depending on policy) → S3 + Postgres.

5 · The hard part — storage tiering and the retention sweep

At 1.8 PB and growing, storage is the dominant cost. Three patterns earn their keep:

The expiry sweep

Pastes with expires_at need to disappear on time. Three ways to do it, from worst to best:

1. Scan the whole table at midnight. Hot, single-threaded, blocks the database. Don't.
2. Index on expires_at, sweep every 15 minutes. Pull rows with expires_at < NOW(), delete from S3, soft-delete the row. The standard answer.
3. Time-bucketed expiry queue. Route each paste to a Redis sorted set keyed by hour. The sweeper just pops the expired bucket. No range scans on the relational store. The next-tier answer when expiry is on the hot path.

6 · Caching strategy

Three layers, the same as URL shortener but with a different ratio of work — the edge does much more here because the payload is bigger.

• Edge cache (CDN). 24-hour TTL on read responses. Catches ~85% of all reads. The viral-paste burst (someone tweets a paste link) lands here, not on origin.
• Redis (metadata only). 5-minute TTL on the id → blob_key, expiry, privacy tuple. ~99% hit rate; saves a Postgres lookup on every cache miss at the edge.
• S3 acts as its own cache. The If-None-Match / If-Modified-Since dance lets the read-svc revalidate cheaply. The blob bytes don't go through the read-svc — it returns a signed URL or a 302, depending on privacy.

7 · Abuse, scanning, takedowns

User-generated content invariably attracts abuse. The this design treats this as first-class infrastructure rather than an afterthought.

A flagged paste is soft-deleted (paste returns 451 Unavailable for Legal Reasons), the blob is moved to a quarantine bucket with restricted IAM, and the event lands in a SIEM. Hard delete only after the appeal window (typically 30 days).

8 · Failure modes & runbook

9 · Cost & SLOs

SLOs

• Read availability: 99.99%. Edge + cross-region replication → 12 min/quarter budget.
• Create availability: 99.9%. Tighter budget; ~2 hours/quarter. Postgres failover dominates.
• Read P99: 200 ms (cache hit) / 500 ms (origin). Track separately; the cache miss rate is the main lever.
• Expiry SLA: 99% within 30 minutes of expires_at. Sweep cadence + CDN TTL bounds.

10 · Trade-offs & "what would you change at 10×"

Further reading

• AWS — "Amazon S3 storage classes & lifecycle". The reference doc on tiered storage costs. Memorise the four standard tiers.
• Cloudflare — "Workers, R2 and the egress-cost model". Useful counter-design — R2 has no egress fees, which changes the optimal CDN architecture for pastebin-shaped workloads.
• Microsoft — "PhotoDNA Cloud Service". The standard CSAM-detection plumbing for any user-generated-content service.
• Backblaze — "How long do hard drives last". Tangential, but useful to internalise that "object storage is cheap" is not magic — it's deduplication, erasure coding, and the bathtub curve.
• Adjacent: URL shortener. Same shape, different storage choice. Read both back-to-back.
• Adjacent: CDNs. The edge layer that makes pastebin economically possible.
• Adjacent: Object storage. The S3-shaped substrate underneath.