Designing a Financial/Payment System
A worked architecture where every choice this journey has covered gets picked for one reason: money cannot silently disappear or duplicate.
Overview
This chapter and the four that follow apply everything this journey has covered to full, worked architectures — real systems with real constraints, where every storage decision has to be justified. A payment system is the strictest case: it's the one architecture in this journey where 'eventually consistent' is never an acceptable answer for the core ledger.
Why It Exists
A social feed can show a like count that's a few seconds stale with zero real consequence. A payment system cannot show an account balance that's wrong, even briefly, without real financial and legal exposure. That single constraint is why this system needs ACIDAtomicity, Consistency, Isolation, Durability — the relational database transaction guarantee.Learn more transactions rather than BASEBasically Available, Soft State, Eventual Consistency — the NoSQL consistency model, trading strict guarantees for horizontal scale.Learn more at every layer. This chapter exists to walk through what that one non-negotiable requirement — money must be exactly right, always — forces at every layer: idempotency keys on every request, and audit logs that are physically impossible to alter after the fact.
Real World Example
A customer clicks 'Pay $49.99' and their network drops before the response arrives, so their browser automatically retries the request. Without an idempotency key, this charges the customer twice — the second request looks, from the server's perspective, like a completely independent, legitimate charge. With an idempotency key generated once by the client and sent on both the original and retried request, the server recognizes the second request as a duplicate and returns the original result without charging again.
The Architecture, Layer by Layer
The Ledger — Double-Entry Bookkeeping as the Source of Truth
Every money movement is recorded as a balanced pair of debit and credit entries in a relational database, never as a single mutable 'balance' field — this makes the full transaction history reconstructable and auditable, not just the current state.
ACID Transactions for Every Money Movement
Debiting one account and crediting another happens inside a single ACID transaction (see the Transactions & Consistency Theory path) — if either side fails, the whole operation rolls back, so money is never deducted from one account without appearing in the other.
Idempotency Keys — Preventing Duplicate Charges
Every payment request carries a client-generated idempotency key; the server stores completed keys and returns the original result for any repeat, rather than re-processing a retried request as new.
Immutable Audit Logs via S3 Object Lock
Every transaction is also written to an append-only audit log stored with S3 Object Lock (WORM mode), so that even a compromised application server or a malicious insider cannot alter the historical record after the fact.
Diagram
A payment request, from idempotency check to immutable audit trail
Payment request + idempotency key
Key seen before?
yes → return original result
ACID transaction: debit + credit
atomic — both happen or neither does
Append-only audit log
S3 Object Lock (WORM) — cannot be altered
Common Mistakes
Storing account balance as a single mutable field updated in place, with no transaction history
Why: A single overwritten balance field can't be audited or reconciled after the fact — if it's ever wrong, there's no record of how it got that way.
Fix: Use double-entry bookkeeping, deriving the current balance from the sum of historical ledger entries rather than storing it as the only source of truth.
Skipping idempotency keys because 'the client shouldn't retry a payment request'
Why: Network failures, client-side retries, and load balancer failovers happen regardless of what the client is supposed to do — without an idempotency key, any of these can cause a real duplicate charge.
Fix: Require an idempotency key on every state-changing payment request, and reject or dedupe requests without one.
Storing audit logs in a normal, mutable database table
Why: A normal table can be edited by anyone with sufficient database access, including a compromised application or a malicious insider — undermining the entire point of an audit trail.
Fix: Write audit logs to storage that's technically incapable of modification, like S3 Object Lock in WORM mode, not just access-controlled.
Interview Questions
Why does a payment system's core ledger need ACID transactions rather than eventual consistency?
Because a temporarily incorrect account balance has real financial and legal consequences, unlike, say, a temporarily stale like count on a social post — the ledger needs a guarantee that a debit and its matching credit either both happen or neither does, which is exactly what an ACID transaction provides.
How does an idempotency key prevent a retried payment request from charging a customer twice?
The client generates a unique key once per logical payment attempt and sends it with the request. The server checks whether it has already processed that key — if so, it returns the original result without re-executing the charge, so a network-level retry of the same request is recognized as a duplicate rather than treated as a new, independent charge.
Why is 'access-controlled' not the same guarantee as 'immutable' for a financial audit log, and how would you actually enforce immutability?
Access control (permissions, roles) determines who is allowed to modify data, but doesn't prevent modification by anyone who does have sufficient access — a compromised admin credential or a bug in an internal tool can still alter or delete records in an access-controlled but otherwise ordinary table. True immutability means the storage layer itself refuses modification regardless of who's asking, which is what S3 Object Lock's WORM mode enforces at the storage layer — not even root-level access to that bucket can alter an object once it's locked for its retention period. I'd write every transaction to WORM storage as an independent, append-only record, in addition to (not instead of) the queryable ledger database.
Production Best Practices
Do
✓Model the ledger as immutable, balanced double-entry records, not a single mutable balance field.
✓Require an idempotency key on every state-changing payment request.
✓Write audit logs to genuinely immutable storage (e.g. S3 Object Lock/WORM), not just access-controlled tables.
Don't
✗Don't store a payment system's balance as a single field with no historical record.
✗Don't assume clients won't retry requests — design for retries happening regardless.
✗Don't treat access control on a database table as equivalent to true immutability.
Comparison
| Consistency Model | Auditability | Best Fit | |
|---|---|---|---|
| Single-region relational DB (Postgres) | Strong (ACID) | High, with double-entry design | Most payment systems at normal scale |
| Distributed SQL (Spanner/CockroachDB) | Strong (ACID) at global scale | High, with double-entry design | Global-scale payment systems needing multi-region consistency |