Event ingestion

1 · Clarifying questions

Five minutes. The correctness bar is the question that changes the architecture more than any other — pin it down first.

2 · Capacity math, on a napkin

Storage is the dominant cost line; the math here decides whether the design is affordable. Run it before drawing boxes.

6 PB/year of raw bytes is what shapes the architecture. Without 10× columnar compression the design isn't affordable. Without tiering (hot → warm → cold) the design isn't affordable at 5 years.

3 · API and data model

Batch ingest at the edge. Three storage shapes, each serving a different read pattern.

Endpoints

POST /v1/events                # batch ingest from advertiser SDKs / pixel
{"events":[
  {"event_id":"01HXY...", "advertiser_id":"A123", "campaign_id":"C9",
   "click_ts":1715990400123, "user_hash":"sha256:...",
   "ip":"203.0.113.7", "ua":"Mozilla/...", "referrer":"...",
   "fraud_signals":{"bot_score":0.04, "vpn":false}}
]}
→ 202 {"accepted":N, "rejected":[{"event_id":"...", "reason":"schema"}]}

GET /v1/reports/clicks         # customer-facing rollups
?campaign_id=C9&from=2026-05-17T00&to=2026-05-18T00&granularity=hour
→ 200 {"buckets":[{"ts":"2026-05-17T00", "clicks":12044, "billable":11890}, ...]}

GET /v1/reports/top            # top-K within window
?advertiser_id=A123&metric=spend&k=10&window=24h
→ 200 {"top":[{"campaign_id":"C9", "spend":1240.55}, ...]}

Schemas

-- raw events (immutable, append-only, columnar)
events_raw
  event_id      ULID PRIMARY KEY        -- client-supplied; idempotency key
  advertiser_id VARCHAR
  campaign_id   VARCHAR
  click_ts      TIMESTAMP(3)            -- ms precision
  ingest_ts     TIMESTAMP(3)            -- server time, for watermarking
  user_hash     CHAR(64)
  ip            INET
  ua            TEXT
  referrer      TEXT
  fraud_signals JSONB                   -- bot_score, vpn, ip_reputation, ...
  partition_key UINT32                  -- hash(campaign_id) % 128
  PARTITION BY toDate(click_ts), partition_key   -- ClickHouse-style

-- aggregated rollups (the customer query target)
clicks_by_hour
  campaign_id   VARCHAR
  hour          TIMESTAMP               -- truncated to hour, in UTC
  clicks_total  UINT64                  -- including suspected fraud
  clicks_billable UINT64                -- post fraud filter
  spend_micros  UINT64                  -- billable × CPC, in micro-units
  PRIMARY KEY (campaign_id, hour)

-- hot cache (Redis), TTL ~ 1 hour
rollup:<campaign_id>:<hour>  →  {clicks_total, clicks_billable, spend_micros}

The event_id is the idempotency key — client-generated, ULID for time-ordering. Consumers dedupe on it. The partition_key spreads a single hot campaign across multiple Kafka partitions; without it, one viral campaign starves every other consumer on that partition.

4 · High-level architecture

Edge in front, Kafka in the middle, two-tier storage at the back. The real-time path gets the customer their numbers within a minute; the batch path corrects them.

Edge validates and timestamps. Kafka is the durable spine — every event lands here before any storage. The realtime consumer maintains Redis rollups for the dashboard; the batch job is the source of truth for billing. Fraud signals flow in asynchronously and update the batch rollups.

5 · Deep dive — exactly-once accounting

This is the part most candidates fumble. "Exactly-once" as a delivery guarantee is famously contentious; the production pattern is more honest about what's actually achievable.

Kafka gives at-least-once delivery by default — a consumer can crash after processing a message but before committing its offset, and the message will be redelivered. The accepted answer is at-least-once delivery plus idempotency at the consumer. The client generates an event_id (a ULID); the consumer's write is keyed on it; a duplicate write is a no-op. The result is exactly-once effects, regardless of how many times the message is delivered.

For billing-grade correctness, idempotency at one tier isn't enough. The pattern is two-tier reconciliation: the real-time path is approximate and optimised for freshness; the batch path is authoritative and optimised for correctness. The customer dashboard shows the real-time number, usually within a few percent of the truth. A scheduled batch job (hourly or daily) recomputes the rollups from the raw event log and overwrites the rollup table. Customer-visible numbers shift slightly when the batch runs; that shift is what gets invoiced.

Concretely: the consumer does a conditional write into ClickHouse keyed on event_id (or maintains a Bloom filter + lookup table of recently-seen IDs). The batch job runs SELECT COUNT(DISTINCT event_id) ... GROUP BY campaign_id, hour from the raw table and overwrites clicks_by_hour. The dashboard reads from clicks_by_hour with a fallback to Redis for the current hour.

6 · Failure modes & runbook

Each of these has bitten a real ad network. The fixes are not exotic; the interesting part is recognising the failure before it hits production.

7 · Cost & operability

Storage dominates. The ingest tier is CPU- and network-bound but cheap relative to the petabytes of retained data. A ballpark in 2026 cloud prices:

Tiering policy

• 7 days hot. ClickHouse, sub-second queries, full row detail.
• 90 days warm. ClickHouse + Parquet on S3 Standard, queries via the same engine, second-class latency.
• 5 years cold. Parquet on S3 Glacier or equivalent. Pre-aggregated to day granularity. Retrieval is async; customer-facing queries beyond 90 days return "ready in N minutes" and email a link.

8 · Trade-offs & what's next

The last five minutes of the interview. The questions below are the ones a strong interviewer reaches for once the core design holds together.

Further reading

• Jay Kreps — "Questioning the Lambda Architecture" (2014). The argument for Kappa, written by the author of Kafka. Worth reading before defending either choice in an interview.
• Confluent — "Exactly-Once Semantics in Apache Kafka". The most precise public account of what Kafka's "exactly-once" actually means and where the boundaries are.
• Google — "MillWheel: Fault-Tolerant Stream Processing at Internet Scale" (2013). Where the watermark idea was first written down clearly.
• ClickHouse — "Sparse Primary Indexes" docs. Why columnar stores can answer aggregation queries at the speeds they do.
• Stripe — "Idempotency Keys at Stripe". The pattern for client-generated idempotency keys; applies cleanly to event ingest.
• Adjacent: How Kafka works. The mechanics under the ingest tier.
• Adjacent: Consistent hashing. Background for the partition-key choice.
• Adjacent: Notifications playbook. Same Kafka-tier shape, different correctness bar.