Design a transaction processing system

Reference Answer

For a full example answer with detailed architecture diagrams and deep dives, see our Design Payment System guide. The payment guide covers ledger design, idempotent transaction processing, and saga patterns that directly apply to a deposit and withdrawal system.

Also review the Databases and Message Queues building blocks for background on ACID-compliant storage and event-driven transaction workflows.

Problem Statement

Design a transaction processing system that allows users to perform withdrawals and deposits, ensuring data consistency, security, and reliable transaction handling. Think of the ledger engine inside Cash App, Venmo, or a bank's mobile application that must never lose money, double-spend, or show an incorrect balance.

The core challenge is maintaining correctness under concurrency, failures, and retries. When two requests hit the same account simultaneously -- a deposit and a withdrawal -- the system must guarantee that the resulting balance is accurate and that neither operation is lost or duplicated. You need to reason about atomicity, isolation, idempotency, and auditability, and design a write path that scales horizontally without sacrificing per-account consistency. Interviewers expect you to discuss tradeoffs between latency and strict guarantees, handle hot accounts that receive many concurrent transactions, and ensure durable event-driven integration with downstream systems like notifications and analytics.

Key Requirements

Functional

• Deposits and withdrawals -- users can deposit funds and withdraw with real-time validation of sufficient balance, receiving a clear success or failure outcome
• Accurate balance -- users see an up-to-date balance that reflects all completed transactions, derived from an authoritative ledger rather than a mutable counter
• Transaction history -- users can view an ordered, immutable history of all transactions on their account at any time
• Authentication and authorization -- users authenticate securely and authorize transactions with appropriate safeguards such as multi-factor authentication

Non-Functional

• Scalability -- support millions of accounts with thousands of transactions per second, handling bursty patterns during payroll cycles or promotional events
• Reliability -- 99.99% uptime with zero data loss for committed transactions; the system must recover to a consistent state after any failure
• Latency -- complete a deposit or withdrawal in under 200ms at p99, with balance lookups under 50ms
• Consistency -- strong consistency for balance calculations and transaction ordering; no double-spending or phantom reads permitted

What Interviewers Focus On

Based on real interview experiences, these are the areas interviewers probe most deeply:

1. Concurrency Control and Double-Spend Prevention

The heart of this system is ensuring that concurrent operations on the same account never result in incorrect balances. Interviewers want to see how you prevent two simultaneous withdrawals from spending the same money.

Hints to consider:

• Use database transactions with SERIALIZABLE or REPEATABLE READ isolation to prevent phantom reads and lost updates on account records
• Implement optimistic locking with version numbers on the balance projection so concurrent modifications are detected before commit
• For hot accounts receiving hundreds of transactions per second, consider per-account queuing or sharded sub-ledgers that batch and serialize operations
• Address deadlock prevention when a transfer touches two accounts: always lock accounts in a deterministic order (e.g., lower account ID first)

2. Idempotent Transaction Processing

Network failures and client retries are inevitable. Interviewers expect a concrete strategy to prevent duplicate deposits or withdrawals when a request is retried after a timeout.

Hints to consider:

• Require clients to send a unique idempotency key with each transaction; store the key atomically alongside the ledger entry with a unique constraint
• On retry, check the idempotency key table first: if the key exists, return the original result without re-executing the transaction
• Define a TTL policy for idempotency records (e.g., 24 hours) to bound storage while covering realistic retry windows
• Extend idempotency across distributed services by passing the key to downstream calls (notifications, analytics) to prevent duplicate side effects

3. Immutable Ledger and Audit Trail

Financial systems require a complete, immutable history of all state changes. Interviewers want to see an append-only design rather than in-place balance mutation.

Hints to consider:

• Record every deposit, withdrawal, fee, and adjustment as an immutable ledger entry with a monotonic timestamp, never deleting or updating past entries
• Use double-entry bookkeeping where every transaction creates a debit entry on one account and a credit entry on another (or a system account), ensuring the books always balance
• Derive the current balance by summing ledger entries (or maintaining a materialized balance projection updated atomically with each new entry) rather than storing it as a standalone mutable field
• Handle corrections and chargebacks by creating compensating entries that reverse the original transaction, preserving the full audit trail

4. Hot Account Mitigation

Some accounts (merchants, promotional accounts) may receive thousands of transactions per second, creating a serialization bottleneck. Interviewers assess whether you can scale writes for these cases.

Hints to consider:

• Partition hot accounts into sub-ledgers (shards) that process transactions in parallel, periodically reconciling into the master balance
• Use an in-memory queue per account that batches multiple pending transactions into fewer database commits
• Explore asynchronous processing where deposits are acknowledged immediately and posted to the ledger via a durable queue, with the balance eventually catching up
• Monitor for hot accounts dynamically and automatically enable sharding when transaction rates exceed a threshold

Suggested Approach

Step 1: Clarify Requirements

Begin by confirming scope and constraints. Ask about the expected transaction volume, the ratio of deposits to withdrawals, and whether cross-account transfers are in scope. Clarify whether the system needs to integrate with external banking rails or operates as a closed-loop wallet. Verify consistency requirements: can users tolerate brief delays in seeing incoming deposits, or must everything be immediately consistent? Ask about regulatory requirements such as transaction limits, KYC verification, and data retention policies.

Step 2: High-Level Architecture

Sketch the core components: an API Gateway for authentication and rate limiting, a Transaction Service that validates requests and coordinates the processing flow, a Ledger Service that maintains the append-only transaction log in PostgreSQL, a Balance Service that serves materialized balances from a Redis cache backed by the ledger, and a Notification Service for real-time alerts. Introduce Kafka as the event backbone: after a ledger write commits, an event is published for downstream consumers (notifications, analytics, fraud detection). Show the transaction flow: client sends a request with an idempotency key, the Transaction Service validates and acquires a per-account lock, writes to the ledger atomically, updates the balance projection, publishes an event, and returns the result.

Step 3: Deep Dive on Transaction Processing

Walk through a withdrawal in detail. The client sends a withdrawal request with an idempotency key and amount. The Transaction Service checks the idempotency key in Redis (fast path) and then PostgreSQL (authoritative). If the key is new, it opens a database transaction, locks the account row (SELECT FOR UPDATE), reads the current balance projection, validates sufficient funds, writes an immutable ledger entry (debit), updates the materialized balance atomically within the same transaction, stores the idempotency key with the transaction ID, and commits. On success, an event is published to Kafka for notifications and analytics. If any step fails, the database transaction rolls back and the client can safely retry with the same idempotency key. Discuss how optimistic locking with version numbers provides an alternative to SELECT FOR UPDATE that reduces lock contention for accounts with moderate traffic.

Step 4: Address Secondary Concerns

Cover hot account handling: detect accounts exceeding a transaction-rate threshold and dynamically partition their ledger into sub-shards processed by dedicated workers. Discuss disaster recovery: synchronous replication to a standby PostgreSQL instance, point-in-time recovery from WAL archives, and Kafka log replay for rebuilding downstream state. Address monitoring: instrument transaction latency, ledger write throughput, idempotency key hit rate, and lock contention metrics. Mention security: encrypt sensitive fields at rest, enforce TLS for all communication, implement rate limiting per account, and flag anomalous patterns for fraud review. Finally, discuss scaling reads: serve balance queries from Redis with a write-through cache updated on every ledger commit, and use read replicas for transaction history queries.

Related Learning

• Payment System -- complete reference answer for ledger design, idempotency, and saga patterns
• Databases -- PostgreSQL ACID transactions, row-level locking, and partitioning
• Message Queues -- Kafka for event-driven processing and durable integration
• Caching -- Redis for fast balance lookups and idempotency key checks
• Rate Limiters -- protecting hot accounts and preventing abuse