Design Recommendation System

Design a recommendation system for Atlassian’s Confluence that suggests relevant pages, blog posts, and documents to a user in real time as they browse or search. The system must support 50 M+ monthly-active users, 1 B+ content items, and 200 K requests/sec at p99 read latency < 200 ms. Recommendations should be personalized via both collaborative-filtering (user-to-user and item-to-item) and content-based signals (title, labels, text embeddings, project, recent editors, comments, shares, time-spent). Provide explicit handling of the cold-start problem for new users and new pages. The design must separate heavy offline computation (matrix factorization, two-tower deep model training, embedding generation) from lightweight online serving (candidate retrieval, real-time re-ranking with session context). Include a feature store that guarantees training/serving skew < 1 %. Support A/B testing of different algorithms and business rules (diversity, freshness, popularity boost). Outline the data ingestion pipeline, candidate-generation funnel (ANN index), multi-stage ranking (L1 lightweight model, L2 deep cross-encoder), re-ranking with guardrails, caching strategy, and evaluation metrics (CTR, engagement-time, precision@10, NDCG). Discuss how to keep recommendations fresh (incremental model updates within 15 min) and how to store and serve 500 M dimensional embeddings at sub-50 ms lookup. Finally, explain how you would detect and mitigate popularity bias and filter bubbles while injecting serendipity.