CTR Prediction System Design (ML)

Design a real-time Click-Through-Rate (CTR) prediction system for Pinterest that can score every candidate pin in the home-feed and ads slots within the 100 ms request budget. The system must ingest 10+ TB/day of new interaction logs, update models continuously, and serve 100k QPS with p99 latency <10 ms per call. You should cover: (1) feature engineering pipeline that produces user embeddings (interests, short/long-term history), pin embeddings (image, text, category, freshness), and context features (time, device, position, surface); (2) model architecture starting from a two-tower neural network (user tower vs. pin tower) that outputs 64-dim embeddings, followed by a lightweight interaction layer producing a calibrated CTR probability; (3) training pipeline that handles 5 B examples/day, supports incremental/online learning, and prevents label leakage; (4) serving infrastructure including feature store, model store, GPU/CPU mixed inference, caching, and A/B testing; (5) calibration and business constraints: predicted CTR must be well-calibrated (RMSE ≤ 0.005 on binned predictions), support multi-objective optimization (CTR, repin-rate, ad revenue), and respect advertiser budgets and pacing. Discuss how you would evaluate offline (AUC, cross-entropy, calibration curve) and online (bucket-level CTR, revenue, user engagement), and how to handle cold-start users and new pins.