Practice/Microsoft/Design a Circuit Breaker

Design a Circuit Breaker

Object Oriented DesignMust

Problem

Design and implement a rate-limited API client that controls the frequency of outbound requests to prevent overloading external services. Your implementation should use the token bucket algorithm to enforce rate limits while allowing controlled bursts of traffic.

The client must track available request capacity, automatically refill tokens over time, handle concurrent requests safely, and provide visibility into usage patterns through metrics. This is a common pattern used in production systems to ensure reliability when interacting with third-party APIs that have rate limits.

Requirements

1. Implement a token bucket algorithm that refills at a configurable rate per second
2. Support configurable maximum bucket capacity to allow controlled bursts
3. Block or throttle requests when tokens are unavailable
4. Ensure thread-safety for concurrent request handling
5. Track and expose metrics including total requests, throttled requests, and available capacity
6. Automatically refill tokens based on elapsed time since last refill
7. Handle edge cases like initial burst capacity and token accumulation limits

Constraints

• Rate limit: 1 to 1000 requests per second
• Bucket capacity: 1 to 10000 tokens maximum
• The system should handle concurrent requests from multiple threads
• Time precision should be at millisecond level or better
• Metrics must be accurate and thread-safe

Examples

Example 1: Basic rate limiting

` client = RateLimitedAPIClient(rate_limit=2, bucket_capacity=2)

First 2 requests succeed immediately

response1 = client.make_request("/api/users") # Success response2 = client.make_request("/api/posts") # Success

Third request blocks/throttled (no tokens available)

response3 = client.make_request("/api/comments") # Throttled

After 0.5 seconds, 1 token refilled

time.sleep(0.5) response4 = client.make_request("/api/likes") # Success `

Example 2: Burst handling

` client = RateLimitedAPIClient(rate_limit=5, bucket_capacity=10)

Can make 10 rapid requests (burst capacity)

for i in range(10): response = client.make_request(f"/api/data/{i}") # All succeed

11th request throttled

response = client.make_request("/api/data/10") # Throttled

metrics = client.get_metrics()

{

"available_tokens": 0,

"total_requests": 11,

"throttled_requests": 1

}

`

Example 3: Token refill over time

` client = RateLimitedAPIClient(rate_limit=3, bucket_capacity=5)

Use all 5 tokens

for i in range(5): client.make_request(f"/api/endpoint/{i}")

Wait 2 seconds (6 tokens would refill, but capped at 5)

time.sleep(2)

Can make 5 more requests

for i in range(5): client.make_request(f"/api/endpoint/{i}") # All succeed `