Design Inference API System

Design a high-concurrency inference API for LLM serving.

Requirements:

• Handle thousands of concurrent requests
• Batch requests for GPU efficiency
• Support multiple user tiers (free/paid/enterprise)
• Auto-scaling based on load
• Rate limiting when overloaded

Key Components:

1. Load Balancer - Distribute traffic across API servers
2. Rate Limiter / API Gateway - Check API keys, limit by tier
3. Message Queues (Redis) - Separate queues per tier
4. Request Batcher - Group requests (wait for 32 OR 40ms timeout)
5. GPU Worker Pool - Process batches
6. Response Handler - Match results to waiting clients

Batching Strategies:

• Fixed Batch Size: Wait for 32 requests (efficient but slow for low traffic)
• Timeout-Based: Wait for 32 OR 40ms (fast but may send small batches)
• Adaptive: Change batch size based on traffic

Calculations:

• Target: 1000 RPS
• Batch size: 32 requests
• Inference time: 50ms/batch
• Overhead: 20ms (grouping) + 5ms (data transfer) = 75ms total
• Batches/GPU/sec: 1000ms / 75ms = 13.3
• Requests/GPU/sec: 13.3 × 32 = 426
• GPUs needed: 1000 / 426 / 0.7 ≈ 4 GPUs at 70% utilization

Discussion Points:

• Why 70% utilization? (headroom for spikes)
• Cold start handling (GPUs take 1-5 min)
• Rate limiting during partial outages
• Caching for repeated prompts