Design a price notification system

Problem Statement

Design a price tracking and notification service that lets users monitor Amazon product prices, subscribe to drop alerts, and view historical pricing trends. Users add products by URL or ASIN, set target-price or percentage-drop rules, and receive timely notifications through their preferred channels when conditions are met. The platform periodically fetches current prices, stores a time-series history, and renders charts showing price movement over configurable windows.

The system must handle tens of millions of tracked products, schedule and execute crawls at scale while respecting source rate limits, detect meaningful price changes without flooding users with noise from minor fluctuations, and fan out notifications to millions of subscribers for popular items. Interviewers use this problem to test your ability to design ingestion pipelines, event-driven notification workflows, high-fanout delivery, and time-series storage, with emphasis on scheduling, deduplication, idempotent alerting, and cost management.

Key Requirements

Functional

• Product tracking -- Users add Amazon products by URL or ASIN to a personal watchlist and see the current price immediately
• Alert subscriptions -- Users define notification rules such as target price thresholds, percentage drops, or any-change triggers, with per-product granularity
• Price history visualization -- Users view historical price charts across configurable time windows (7 days, 30 days, 1 year) for any tracked product
• Notification preferences -- Users choose delivery channels (email, push, digest) and can pause or resume alerts per product or globally

Non-Functional

• Scalability -- Track 50 million products with price checks distributed across configurable intervals; support 10 million subscribers with peak notification fan-out of 100,000 messages per second
• Reliability -- Guarantee at-least-once notification delivery with idempotent deduplication; tolerate crawl failures and source throttling without data gaps
• Latency -- Deliver alerts within 5 minutes of detecting a qualifying price change; serve price history queries in under 500 ms
• Consistency -- Eventual consistency for price updates appearing in history charts; strong consistency for subscription management to prevent missed alerts

What Interviewers Focus On

Based on real interview experiences, these are the areas interviewers probe most deeply:

1. Scalable Price Collection Pipeline

Fetching prices for tens of millions of products requires a well-orchestrated crawl system that respects source rate limits, prioritizes frequently changing items, and handles failures gracefully.

Hints to consider:

• Use a job scheduler that distributes crawl tasks across worker pools with configurable intervals per product based on popularity and price volatility
• Implement per-domain rate limiting with token bucket algorithms to avoid being throttled or banned by the source
• Apply adaptive scheduling: increase crawl frequency for products with recent price movement and decrease it for stable items
• Handle transient failures with exponential backoff and a dead-letter queue for items that repeatedly fail, with alerts to operators

2. Change Detection and Alert Evaluation

Not every price observation warrants a notification. Minor fluctuations, duplicate observations, and rounding differences must be filtered out before evaluating subscriber rules.

Hints to consider:

• Compare each new price against the last confirmed price using a meaningful threshold (absolute or percentage) to filter noise
• Store the latest confirmed price per product in a fast cache (Redis) for instant comparison without hitting the time-series store
• When a meaningful change is detected, publish a price-change event to a stream (Kafka topic) that downstream consumers evaluate against subscriber rules
• Implement idempotent change detection using observation IDs so duplicate crawl results do not trigger duplicate events

3. High-Fanout Notification Delivery

A popular product can have millions of subscribers. When its price drops, the system must fan out notifications without overwhelming downstream providers or creating duplicate sends.

Hints to consider:

• Decouple the price-change event from individual notification dispatch using a two-phase pipeline: first expand the subscriber list, then batch-deliver per channel
• Shard subscriber expansion by product partition or subscriber segment to parallelize the work
• Use per-provider rate limiting and circuit breakers to protect email, SMS, and push services from overload
• Assign a deterministic notification ID per (subscriber, product, event) tuple and check it against a Redis cache with TTL to guarantee exactly-once delivery

4. Time-Series Price Storage and Querying

Historical price data grows continuously and must support both fast recent-window queries for user-facing charts and efficient long-term retention for trend analysis.

Hints to consider:

• Partition price observations by product_id and time bucket for efficient range scans
• Use a time-series-friendly store (DynamoDB with sort key on timestamp, TimescaleDB, or ClickHouse) that supports fast range reads and automatic compaction
• Downsample older data (keep hourly averages after 90 days, daily after one year) to control storage costs while preserving trend visibility
• Cache recent price windows for popular products in Redis to serve chart queries without hitting the database

5. Crawl Scheduling and Cost Management

Crawling millions of products is expensive in compute and network terms. Interviewers want to see intelligent prioritization that maximizes freshness within a budget.

Hints to consider:

• Assign a priority score to each product based on subscriber count, recent price volatility, and time since last successful crawl
• Use a priority queue (Redis sorted set or a scheduler like Airflow) that dynamically reorders crawl tasks
• Set a global crawl budget (requests per minute) and allocate capacity proportionally across priority tiers
• Monitor crawl success rate, average staleness, and budget utilization to tune scheduling parameters over time

Suggested Approach

Step 1: Clarify Requirements

Confirm the number of tracked products, subscriber scale, and expected notification volume. Ask whether the system must support sources beyond Amazon and whether product data beyond price (reviews, ratings) is in scope. Clarify acceptable staleness for price data and latency targets for alert delivery. Determine whether digest-style notifications (daily summary email) are required alongside real-time alerts. Ask about compliance concerns such as data retention policies or user data deletion rights.

Step 2: High-Level Architecture

Sketch the major components: a product and subscription API backed by PostgreSQL for user accounts, watchlists, and alert rules. A crawl scheduler that publishes crawl tasks to a Kafka topic, consumed by a pool of fetcher workers that respect rate limits. A change detection service that compares fetched prices against the cached last-known price and publishes price-change events. A notification pipeline that expands subscriber lists, evaluates rules, deduplicates, and dispatches through channel-specific providers. A time-series store (DynamoDB or TimescaleDB) for price history. A Redis layer for the latest price cache, deduplication keys, and rate-limiting counters.

Step 3: Deep Dive on Change Detection and Notification Flow

Walk through the path from a crawl result to a delivered alert. A fetcher worker retrieves the current price for a product, publishes the raw observation to a Kafka topic. A change detection consumer reads it, loads the last confirmed price from Redis, and compares. If the change exceeds the noise threshold, it publishes a price-change event to a separate topic. A notification expansion consumer reads this event, queries the subscriber list for the product, evaluates each subscriber's rules (target price, percentage drop), and for matching subscribers, produces individual notification messages onto channel-specific topics (email, push). A delivery worker picks up each message, checks the deduplication cache, formats the content, and calls the provider API. On success, it writes a delivery record; on failure, it retries with backoff up to a maximum before routing to the dead-letter queue.

Step 4: Address Secondary Concerns

Cover crawl resilience: fetcher workers implement per-domain rate limits, exponential backoff on failures, and circuit breakers that pause crawling a source when error rates spike. Discuss storage lifecycle: raw observations are retained for 30 days at full resolution, then downsampled to hourly aggregates, and eventually to daily aggregates after one year. Address monitoring: track crawl throughput, failure rate by source, average data staleness, notification delivery rate, and duplicate suppression counts. Mention scalability: partition Kafka topics by product_id, scale fetcher and notification workers independently based on queue depth, and shard Redis by key prefix. Briefly touch on security: encrypt stored credentials, use OAuth for any authenticated source access, and rate-limit subscription creation to prevent abuse.