Design a Secure Financial Data Pipeline

Design a secure, compliant, and highly-available data pipeline that ingests millions of financial transactions per day from Plaid’s banking and fintech partners, transforms the data for downstream analytics and reporting, and serves it to both internal risk systems and external customer APIs. Your system must guarantee end-to-end encryption, maintain immutable audit logs, and support GDPR & CCPA “right-to-be-forgotten” deletes without breaking immutable transaction history required by U.S. banking regulations. You are expected to handle batch daily loads (≈ 500 GB) and a real-time stream (≈ 20 k events/sec) with < 100 ms end-to-end latency for fraud-detection consumers. Design for 99.99 % availability, multi-region failover, and automatic recovery; estimate storage and compute for three years of 50 % YoY data growth; describe how you would detect and quarantine PII or sensitive authentication tokens accidentally emitted by a source; and provide a rollback strategy that can replay the last 24 h of ingestion without duplicate writes or data loss. You have 45 minutes: start with requirements, move to high-level architecture, then deep-dive into security, compliance, and scaling trade-offs.