Twitter Data Stock Prediction

Design a production-grade system that ingests the full Twitter fire-hose in real time, detects tweets that mention publicly traded companies (via cashtags like $AAPL or company names), scores the financial sentiment of each tweet, and produces aggregate sentiment signals that can be consumed by quantitative trading strategies. Your end-to-end pipeline must (1) collect tweets with <1 s ingestion latency, filter English language, remove retweets, drop accounts with <100 followers, and flag obvious bots; (2) run sentiment analysis with a domain-tuned FinBERT model, weight the raw polarity by engagement (likes, retweets, replies) and produce per-ticker metrics every 5 minutes: mean sentiment, standard deviation (disagreement), tweet volume, and positive/negative ratio; (3) store raw tweets, enriched features, and aggregates in queryable form for at least two years; (4) expose a low-latency gRPC/REST API that returns the latest sentiment snapshot for any ticker within 100 ms; (5) guarantee at-least-once delivery and exactly-once processing semantics for financial compliance; (6) scale to 500 M tweets/day (≈6 k/s peak) with horizontal scalability and self-healing; (7) provide a real-time dashboard that plots sentiment and volume for the S&P 500 over sliding 5 min, 1 h, and 1 day windows; and (8) include an automated bot/manipulation detector that quarantines coordinated campaigns, pump-and-dump spam, and accounts younger than 7 days that exceed 100 tweets/hr. Discuss your choices of streaming framework, message bus, storage engines, model serving, feature store, and monitoring/alerting, plus how you would back-test signal efficacy and evolve the model without downtime.