Distributed scheduler

1 · Clarifying questions

Scheduler design lives or dies on the semantics. Pin them down before drawing boxes.

2 · Capacity math, on a napkin

The numbers set the floor for the schedule store, the scheduler tier, and the worker tier. Each scales for a different reason.

The thundering-herd line is the one to call out. A naive design that fires every "midnight UTC" cron at exactly 00:00:00.000 turns a 1K-rps system into a 50K-rps spike. Jitter the fire times by a few seconds and the spike flattens.

3 · API and data model

A narrow surface. The schedule is the contract; the firings table is the audit log.

Endpoints

POST /v1/schedules                    # create
{
  "cron_expr": "0 9 * * MON-FRI",
  "timezone": "America/New_York",
  "target_url": "https://acme.com/webhooks/morning-report",
  "payload": {"team": "ops"},
  "idempotency_key": "morning-report-ops"
}
→ 201 {"id": "sch_01HX...", "next_fire_at": "2026-05-19T13:00:00Z"}

GET    /v1/schedules/:id              # inspect
DELETE /v1/schedules/:id              # cancel
GET    /v1/schedules/:id/firings      # history (paginated)

Schema

schedules
  id              ULID PRIMARY KEY
  owner_id        BIGINT NOT NULL
  cron_expr       VARCHAR(120) NOT NULL
  timezone        VARCHAR(40)  NOT NULL    -- IANA, e.g. America/New_York
  target_url      TEXT         NOT NULL
  payload         JSONB        NULL
  idempotency_key VARCHAR(120) NOT NULL    -- forwarded to target on fire
  status          ENUM('active','paused','deleted') NOT NULL
  next_fire_at    TIMESTAMP    NOT NULL    -- precomputed; the hot index
  shard_id        SMALLINT     NOT NULL    -- hash(id) % N

  INDEX (shard_id, next_fire_at) WHERE status = 'active'  -- the only hot read
  INDEX (owner_id, status)

firings
  id              ULID PRIMARY KEY
  schedule_id     ULID NOT NULL
  fire_at         TIMESTAMP NOT NULL       -- the scheduled time
  attempt         SMALLINT  NOT NULL
  status          ENUM('pending','running','succeeded','failed','dead') NOT NULL
  started_at      TIMESTAMP NULL
  finished_at     TIMESTAMP NULL
  result_code     INT       NULL
  result_excerpt  TEXT      NULL

  INDEX (schedule_id, fire_at)
  INDEX (status, fire_at) WHERE status IN ('pending','running')

leases
  shard_id        SMALLINT PRIMARY KEY
  leader_id       VARCHAR(64) NOT NULL
  expires_at      TIMESTAMP   NOT NULL
  fenced_at       TIMESTAMP   NULL         -- monotonic fencing token

Two writes per firing: claim (mark running, set started_at) and finalise (set status, finished_at). The claim is what makes "fire once" work — a second leader trying to claim the same row loses on the optimistic-concurrency check.

4 · High-level architecture

Three tiers, each with its own scaling axis. The scheduler tier reads the index of next_fire_at; the worker tier executes; the durable store is the source of truth that both sides defer to when they disagree.

A shard is the unit of ownership. One scheduler leader per shard scans next_fire_at, claims rows, and pushes onto the fire queue. Workers are stateless and partition-aware — at-least-once delivery from the queue, idempotency at the target.

5 · Leader election, leases, and the missed-tick rule

The deep dive. One leader per shard is what keeps a firing from going out twice in the normal case. The lease is what keeps a dead leader from stalling that shard forever.

Single-leader-per-shard

Each shard has one elected leader via etcd / Zookeeper. The leader holds a lease with a TTL (e.g. 15 s) and renews it every 5 s. Only the leader for shard k scans rows where shard_id = k and claims firings. A non-leader is idle for that shard. This is the cleanest way to avoid double-firing — exactly one process is allowed to issue a claim at any time.

Lease loss and fencing

A leader that GCs for 20 s loses its lease. A new leader takes over the shard. The old leader might still try to enqueue firings it had in-flight before noticing the loss. Fencing tokens — a monotonically increasing number attached to every claim — let the firings table reject writes from a stale leader. Without fencing, the lease is advisory; with it, it's enforced.

Time triggers vs interval triggers

A cron expression resolves to wall-clock times. An interval trigger ("every 5 min from now") resolves to monotonic times. Mixing them is a common source of drift — if the system stores intervals as wall-clock and the host clock jumps, the next fire skews. Store the trigger as either-or, not both.

The missed-tick rule

Spell this out on the page. When the scheduler comes up after an outage, it scans for rows where next_fire_at < now(). The policy is per-schedule:

• Fire-late. Default for most jobs. Enqueue one firing per missed slot up to a cap (e.g. last 10), tagged with a catch_up marker so the target can decide whether to skip.
• Fire-most-recent. Common for "send the daily report". Skip intermediate missed slots; fire once for the most recent missed time.
• Skip. For user-facing notifications. A "9am reminder" delivered at 2pm is worse than not delivering it.

6 · Failure modes & runbook

The honest part. Most of these are about getting at-least-once right without turning it into many-times.

7 · Cost & operability

The shape of the bill, and the levers that move it. The schedule store dominates; the rest scales with what's actually firing, not with what's scheduled.

8 · Trade-offs & what's next

The last 5 minutes. These are the second-order problems that distinguish a one-tenant scheduler from a platform.

Further reading

• Google — "Reliable Cron across the Planet" (2016). The canonical writeup. How Google's internal cron service uses Paxos for leader election and survives datacenter failover. Read this once.
• Airbnb — "Chronos" (deprecated, but the writeup is good). A distributed cron built on Mesos. The lessons on dependencies between jobs are worth skimming even though the project is dead.
• Quartz Scheduler — "Clustering" docs. The Java-world reference. JDBC-backed lease via a row lock; a useful baseline before designing your own.
• Marc Brooker — "Leases, fencing tokens, and you". The clearest treatment of why a lease alone is not enough and what a fencing token buys you.
• Stripe — "Idempotency Keys at Stripe". The pattern the scheduler relies on at the target. Required reading for at-least-once with a clean user experience.
• Adjacent: Idempotency. The deep dive on the dedupe contract.
• Adjacent: Distributed locks. What the lease is, and what it isn't.
• Adjacent: Time and clocks. Why "fire at 2:30 AM" is harder than it looks.