Design Mint

1 · Clarifying questions

2 · Capacity math, on a napkin

The dominant constraint isn't our infrastructure — it's the partner APIs. Each bank has its own rate limit; some are 10 req/sec across all our users. The scheduler is built around respecting those limits while still finishing the nightly sweep on time.

3 · API and data model

Public API

POST /v1/accounts/link                  # initiate bank connection
{ "institution_id": "chase", "credential_type": "oauth" }
→ 200 { "link_token": "lt_...", "expires_at": "..." }
# Client uses link_token in partner SDK; we get a callback with credentials_id

POST /v1/accounts/link/complete
{ "link_token": "lt_...", "credentials_id": "cred_..." }
→ 201 { "account_ids": ["a_...", "a_..."], "sync_started": true }

POST /v1/accounts/:id/sync              # on-demand refresh
→ 202 { "job_id": "j_..." }

GET  /v1/accounts/:id                   # balance + last sync
GET  /v1/accounts/:id/transactions      # paginated
  ?since=2026-05-01&limit=100

GET  /v1/categories
PUT  /v1/transactions/:id/category      # user override

Storage

institutions                    -- one row per supported bank
  institution_id  VARCHAR(64) PK
  name            TEXT
  partner         VARCHAR(32)   -- plaid | finicity | direct
  rate_limit_qps  INT
  oauth_config    JSONB
  status          VARCHAR(16)   -- live | degraded | down

credentials                     -- encrypted secrets per user×institution
  credentials_id  UUID PK
  user_id         BIGINT
  institution_id  VARCHAR(64)
  encrypted_token BYTEA         -- envelope-encrypted; KMS-managed key
  encryption_kid  VARCHAR(64)
  status          VARCHAR(16)   -- active | expired | revoked

accounts                        -- one row per connected account
  account_id      UUID PK
  user_id         BIGINT
  credentials_id  UUID
  type            VARCHAR(16)   -- checking | savings | credit | brokerage | loan
  partner_id      TEXT          -- partner's id; UNIQUE with credentials_id
  balance         DECIMAL(18,2)
  available       DECIMAL(18,2)
  currency        CHAR(3)
  last_synced_at  TIMESTAMP
  next_sync_at    TIMESTAMP     -- scheduler index
  status          VARCHAR(16)
  INDEX (next_sync_at, status)  -- drives the scheduler

transactions                    -- partitioned by user_id, then date
  txn_id          UUID PK
  account_id      UUID
  partner_txn_id  TEXT          -- UNIQUE (account_id, partner_txn_id)
                                -- the idempotency key
  amount          DECIMAL(18,2)
  currency        CHAR(3)
  posted_at       DATE          -- partition key suffix
  description     TEXT
  merchant        TEXT
  category_id     VARCHAR(32)
  user_category   VARCHAR(32)   -- user override; takes precedence
  INDEX (account_id, posted_at DESC)

sync_jobs                       -- audit trail
  job_id          UUID PK
  account_id      UUID
  started_at, finished_at TIMESTAMP
  status          VARCHAR(16)   -- queued | running | success | failed
  error_class     VARCHAR(64)
  txns_added      INT

4 · High-level architecture

Scheduler scans accounts where next_sync_at <= now and enqueues sync jobs to a queue partitioned by institution_id. Each institution has a dedicated worker pool sized to its rate limit — overflow waits in queue rather than overwhelming the partner. Workers fetch new transactions, dedupe by partner_txn_id, write to Postgres, then categorise. Credentials are envelope-encrypted with KMS keys; workers decrypt on demand and never log plaintext.

5 · The hard part — idempotent sync against fragile partners

Why idempotency is hard here

A sync run pulls transactions, writes them to our DB, then updates last_synced_at. If the worker crashes after writing some transactions but before updating last_synced_at, the next run will pull the same transactions again. Without idempotency we either lose data (skip-ahead) or double-count it (re-insert).

The idempotency key is the partner's transaction ID — every bank gives one back. The transactions table has a unique index on (account_id, partner_txn_id). Re-inserts become no-ops. Updates use an UPSERT pattern:

INSERT INTO transactions
  (txn_id, account_id, partner_txn_id, amount, posted_at, description, ...)
VALUES (...)
ON CONFLICT (account_id, partner_txn_id)
DO UPDATE SET
  amount = EXCLUDED.amount,
  description = EXCLUDED.description,
  posted_at = EXCLUDED.posted_at,
  updated_at = now()
RETURNING txn_id;

Pending vs posted

Most banks return pending transactions that later turn into posted transactions with a different (or sometimes the same!) partner ID. The system has to recognise that "Pending: STARBUCKS $4.20" and "Posted: STARBUCKS $4.20 2026-05-21" are the same transaction. Heuristics: same account, same amount, same merchant string, posted-date close to pending-date. When a posted transaction matches a pending one, the pending row is deleted and the posted one wins.

Rate-limit-aware scheduling

One bank with 10 req/sec across all our users and 5M of our users on that bank → finishing the nightly sweep takes 5,000,000 / 10 / 86,400 ≈ 5.8 days, which doesn't fit a 6-hour window. The fix: stagger refresh frequency by institution. High-rate-limit banks (Chase, BofA, Wells Fargo) refresh nightly. Low-rate-limit small banks refresh every 3 days; on-demand refresh always works, regardless.

The scheduler maintains a per-institution token bucket. Workers acquire a token before each partner call, sleep if the bucket is empty. The bucket replenishes at the institution's rate limit. Centralised in Redis so all worker instances share the same view.

Retry classification

6 · Categorisation

Each transaction gets a category (Groceries, Rent, Dining, Transport, …). Three layers:

• User override. If the user has corrected this merchant before, use that. Saved per (user, merchant) with a normalised merchant string.
• Global rules. Maintained list mapping merchant strings to categories ("STARBUCKS" → Dining). Curated centrally, updated weekly.
• ML model. Trained on aggregate categorised data. Runs only on transactions that miss the override and rules. Outputs category + confidence; below threshold, the transaction is left "Uncategorised" until the user labels it.

Privacy: the model is trained on aggregated, anonymised data. Per-user behaviour is never used for cross-user inference. Compliance audits this regularly.

7 · Failure modes & runbook

8 · Cost & SLOs

SLOs

• Dashboard load P99: 500 ms. Cached aggregates per user; recomputed on transaction insert.
• On-demand sync P99: 30 s. User taps Refresh; we expect to be done in 30 s for top-10 banks; longer for niche institutions.
• Nightly sweep completion: 100% of accounts by 8 AM local. Hard SLO; missing it means stale dashboards at morning peak.
• Transaction freshness P99: 24 h. Bank-dependent; some banks publish only daily.
• Data durability: 11 nines. Multi-region Aurora; daily backups; quarterly restore drills.
• Zero plaintext credential persistence. Verified by quarterly audit.

9 · Trade-offs & "what would you change at 10×"

Further reading

• Plaid Engineering — public posts on auth, transactions, identity verification. The clearest reference for the aggregator model this design assumes.
• "Idempotent APIs" — Stripe Engineering. The canonical writeup on the idempotency-key pattern.
• Truelayer / Tink engineering blogs. European open-banking-flavoured versions of the same design.
• Adjacent: Distributed scheduler. The scheduling layer in depth.
• Adjacent: Idempotence. Why retries are safe.
• Adjacent: OAuth / OIDC. The credentials story.
• Adjacent: Napkin math. The capacity math.