System Design - Design YouTube

Designing an interview question for OpenAI requires blending traditional System Design with the specific constraints of AI infrastructure (e.g., GPU scarcity, high-latency inference, and token streaming).

Problem Statement: Design a Multi-Tenant AI Inference Gateway

The ContextOpenAI needs to manage high-volume API requests for models like o1 or GPT-5 across diverse customer tiers. Your task is to design an AI Gateway that sits between users and a massive cluster of inference nodes.

Functional Requirements

1. Request Handling: Accept prompts and return model-generated tokens via server-sent events (streaming).
2. • Credits must be deducted based on the number of input/output tokens.
   • Credits have expiration timestamps and must be consumed in a "First-to-Expire" order.
3. Credits must be deducted based on the number of input/output tokens.
4. Credits have expiration timestamps and must be consumed in a "First-to-Expire" order.
5. Tiered Prioritization: Support multiple priority queues (e.g., Free, Plus, Enterprise).

Non-Functional Requirements

• Low Latency: The "Time to First Token" (TTFT) must be minimized.
• Accuracy: Credit balances must be eventually consistent across a global infrastructure to prevent overages.
• High Availability: The system must handle millions of concurrent users even if a specific GPU cluster goes offline.

Step-by-Step Design Guidance

1. Define the Data Structure for Credits

To handle credits with expiration, a simple integer balance is insufficient. Use a list of "credit blocks," each with an amount and an expiration date.

Total Balance(t)=∑i=1namountiwhere expirationi>tTotal Balance open paren t close paren equals sum from i equals 1 to n of amount sub i space where expiration sub i is greater than tTotal Balance(𝑡)=𝑛𝑖=1amount𝑖where expiration𝑖>𝑡

A priority queue or a sorted list is typically recommended for this in interviews.

2. Architect the High-Level System

The system should consist of three main layers, which can be visualized through standard system design patterns:

• API Gateway: Handles authentication, rate limiting, and initial request validation.
• Orchestration Layer: Manages the priority queue and selects the best available inference node (GPU cluster).
• Inference Cluster: The actual workers running the LLM.

3. Deep Dive: Handling GPU Scarcity

In an OpenAI-style interview, you must address what happens when demand exceeds GPU capacity. Propose a Batching Service to group smaller requests together to maximize throughput, or explain the trade-offs between "Streaming" vs. "Batching".

4. Address Ambiguity (The "OpenAI Signature")

Interviewer follow-up questions often test how you handle out-of-order events. For example:

• What happens if a 'Spend Credit' event arrives after the balance was already queried at a later timestamp?
• How do you ensure credit deductions are accurate if the inference streaming connection is interrupted mid-response?

Final Answer Summary

To solve this, you must implement a stateful gateway that manages a priority queue of expiring credits while maintaining a low-latency streaming pipeline between the user and the inference nodes. Success is measured by your ability to justify trade-offs between strict credit consistency and system throughput.

Would you like to see a Python class implementation for the GPU credit calculator part of this problem?

[0] - Open AI Coding Interview | Design AI Gateway [1] - I have asked someone from OpenAI to do a System Design ... [2] - OpenAI SWE Coding Interview: GPU Credit Calculator ... [3] - OpenAI SWE Coding Interview: GPU Credit Calculator ... [4] - Design Batch Inference System - Anthropic & OpenAI System ... [5] - OpenAI System Design Interviews (questions, process, prep) [6] - You Can’t Explain Your Project — And That’s Why You Get Rejected