Design Recommendation System

Design a large-scale recommendation system for Atlassian products (Jira, Confluence, Trello) that surfaces relevant work items (issues, pages, cards) to each user in real time. The system must handle 10 M daily active users, 100 M searchable items, and 1 B daily recommendation requests with ≤200 ms p99 latency. Support both collaborative filtering (user-item interaction graphs) and content-based filtering (text embeddings of titles, descriptions, labels). Use a two-tower deep model to produce 64-dim user and item embeddings, store them in an online feature store, and perform approximate nearest-neighbor search for candidate generation. Offline jobs retrain the model nightly and refresh item embeddings; an online layer re-ranks candidates with contextual features (current page, time of day, recent activity). Provide A/B testing infrastructure to measure CTR, dwell time, and NDCG@20. Address cold-start for new users/items via content-based fallback and popularity priors. Ensure freshness by updating embeddings within 15 min of edits and support multi-tenant isolation per Atlassian’s cloud.