Practice/Bloomberg/Design a stock ticker tracking system

Design a stock ticker tracking system

System DesignOptional

Problem Statement

Design a real-time analytics platform that monitors Twitter/X posts to identify and count stock ticker mentions (cashtags like AAPL, TSLA, GOOG), providing aggregated mention counts across multiple rolling time windows (5 minutes, 10 minutes, 30 minutes, and 60 minutes). Users consume the data through REST APIs and a live dashboard to see which tickers are trending, how mention velocity is changing, and what the sentiment breakdown looks like over the past hour.

This problem is tailor-made for Bloomberg, where real-time market data and social sentiment analysis are core to the Terminal product. The platform must handle millions of tweets per hour during peak market events (earnings releases, Fed announcements, market crashes), provide sub-second API response times for ticker queries and top-K trending requests, and maintain count accuracy despite out-of-order tweet delivery, duplicates from API retries, and extreme mention skew when a single ticker goes viral. The fundamental challenge is building a streaming pipeline that ingests, deduplicates, windows, and serves data with tight freshness and latency guarantees while keeping infrastructure costs bounded.

Key Requirements

Functional

• Real-time ticker mention counts -- display accurate mention counts for any tracked stock ticker across four preset rolling time windows (5, 10, 30, and 60 minutes), updating continuously as new tweets arrive
• Top-K trending tickers -- serve queries for the most-mentioned tickers within a chosen time window, ranked by mention count or mention velocity (rate of change)
• Per-ticker detail API -- return current counts, historical sparkline data, and basic sentiment breakdown (positive, negative, neutral) for a specific ticker via a RESTful endpoint
• Configurable tweet filtering -- allow users to include or exclude retweets and replies when viewing counts, and support filtering by geography or language

Non-Functional

• Scalability -- handle 10 million tweets per hour during peak events, track 100,000+ distinct tickers concurrently, and scale processing horizontally by adding consumer instances
• Reliability -- ensure no tweet data is lost during ingestion; survive individual component failures with automated recovery; provide at-least-once processing with idempotent aggregation
• Latency -- tweets should appear in aggregate counts within 2 to 5 seconds of ingestion; API queries must return within 200ms at p99; dashboard updates should reflect new data within 3 seconds
• Consistency -- eventual consistency is acceptable with a 5-second convergence window; counts should never visibly decrease within a time window unless a retraction (tweet deletion) explicitly triggers it

What Interviewers Focus On

Based on real interview experiences, these are the areas interviewers probe most deeply:

1. Streaming Pipeline Architecture

The core challenge is designing an ingestion and processing pipeline that handles variable, bursty load. During earnings season or breaking market news, tweet volume can spike 100x in minutes. Interviewers want to see how you buffer, parallelize, and prevent downstream overload.

Hints to consider:

• Use Kafka as a shock absorber between the Twitter/X API ingestion layer and the stream processing tier, partitioned by ticker symbol or tweet ID hash for parallel consumption
• Deploy Apache Flink for stateful stream processing with event-time windowing, watermarks, and exactly-once state semantics
• Implement backpressure mechanisms so that when processing lags, the ingestion layer slows its pull rate from the Twitter API rather than dropping tweets or crashing workers
• Consider separate processing paths for high-frequency tickers (AAPL, TSLA) versus low-frequency tickers to prevent starvation

2. Windowed Aggregation with Late and Out-of-Order Data

Tweets can arrive delayed due to Twitter API pagination, mobile client offline queuing, or network retries. Interviewers expect you to handle event-time semantics correctly and explain window boundaries.

Hints to consider:

• Use the tweet's original creation timestamp (not processing time) to assign it to the correct time window, ensuring accuracy even when processing is delayed
• Define a watermark strategy that allows a configurable lateness tolerance (e.g., 30 seconds) before closing a window; late arrivals within the tolerance update the aggregate, extremely late arrivals are logged and dropped
• Compute sliding windows efficiently by maintaining 1-minute tumbling window buckets as the base granularity, then composing 5, 10, 30, and 60-minute windows as unions of the appropriate number of 1-minute buckets
• Emit early speculative results before the watermark passes so the dashboard shows approximate counts that refine as more data arrives

3. Hot Key Management and Write Scaling

When a single ticker goes viral (e.g., during a major earnings miss or a meme stock rally), writes to that ticker's aggregation state become a bottleneck. Interviewers look for concrete mitigation strategies.

Hints to consider:

• Implement local pre-aggregation within each Flink task before writing to shared state, batching per-ticker increments over short intervals (e.g., every 500ms) to reduce update frequency
• Use split counters for hot tickers: partition a single ticker's count across N sub-keys (AAPL_shard_0 through AAPL_shard_7) and merge them at read time
• Apply hierarchical aggregation where leaf nodes in the processing topology handle subsets of tweets, intermediate nodes merge partial counts, and the root node produces the final aggregate
• Cache hot ticker aggregates in memory with 1-second TTLs to absorb read spikes without hitting the backend store on every request

4. Serving Layer for Low-Latency Reads

API consumers and dashboards need fast access to current aggregates and top-K rankings. Interviewers want to see separation between the write-optimized streaming state and the read-optimized serving store.

Hints to consider:

• Write finalized window aggregates to Redis using sorted sets (ZADD with mention count as score) for top-K queries and hash maps for per-ticker lookups
• Precompute top-K rankings for each time window and materialize them as sorted lists so serving requires a simple Redis ZREVRANGE rather than a runtime sort
• Use WebSocket or Server-Sent Events connections to push incremental count updates to connected dashboard clients when aggregates change
• Implement a query cache with short TTL (2 to 5 seconds) for frequently requested endpoints to absorb traffic spikes without overloading Redis

Suggested Approach

Step 1: Clarify Requirements

Confirm the problem scope and constraints. Ask about expected peak tweet ingestion rate and how bursty the traffic pattern is. Clarify how many distinct tickers need to be tracked and whether the system must discover new tickers automatically or only track a predefined list. Determine what sentiment analysis complexity is required -- simple positive/negative/neutral classification or more granular scoring with confidence levels. Ask about acceptable latency and accuracy trade-offs: can approximate counts be shown if exact counts would be slower? Confirm whether the system needs to handle tweet deletions by decrementing counts and whether historical queries beyond the 60-minute window are in scope.