Practice/Anthropic/Design a large model file distribution system

Design a large model file distribution system

System DesignOptional

Problem Statement

You are tasked with designing a distribution system that delivers a 500GB machine learning model file from an external storage location to 1,000 servers within your datacenter. The ingress connection to your datacenter is limited to 10Gbps, and you need to minimize the total time required to get a complete, verified copy of the model onto every target server.

This system is similar to a datacenter-internal content delivery network designed specifically for massive ML artifacts. The challenge lies in efficiently converting a single slow ingress stream into a fast, parallel distribution across hundreds of machines without overwhelming the network, corrupting data, or requiring expensive restarts when failures occur. Your design must handle the physics of limited bandwidth, the reality of transient failures, and the need for verifiable data integrity across all recipients.

Key Requirements

Functional

• Distribution initiation -- Users can trigger a deployment job specifying which model version to deploy and which servers should receive it
• Progress monitoring -- Real-time visibility into per-server download status, completion percentage, and overall job health
• Integrity verification -- Every target server must cryptographically verify it received a complete, uncorrupted copy before activating the model
• Resumable transfers -- Interrupted downloads can pick up where they left off without re-downloading completed portions

Non-Functional

• Scalability -- Must efficiently distribute to 1,000+ servers; architecture should extend to tens of thousands
• Reliability -- System must handle server crashes, network partitions, and transient failures without manual intervention
• Latency -- Total distribution time should approach the theoretical minimum given the bandwidth constraints (significantly better than naive sequential copying)
• Consistency -- All servers must receive byte-for-byte identical copies; no server should activate a corrupted or incomplete model

What Interviewers Focus On

Based on real interview experiences, these are the areas interviewers probe most deeply:

1. Bandwidth Optimization and Fanout Strategy

Interviewers want to see if you recognize that a naive approach (fetching from origin 1,000 times) is physically impossible given the 10Gbps constraint. The key insight is converting a single slow ingress into a peer-assisted swarm.

Hints to consider:

• Break the file into manageable chunks (e.g., 64MB or 128MB pieces) so peers can start sharing before the entire file arrives
• Use a tree or swarm topology where servers that have downloaded chunks can immediately become sources for peers
• Calculate theoretical minimums: a single 500GB transfer over 10Gbps takes ~7 minutes; you want total completion in 10-15 minutes, not 7,000 minutes
• Consider rack-aware scheduling to maximize intra-rack bandwidth and avoid cross-rack bottlenecks

2. Data Integrity and Chunk Management

A 500GB file cannot be treated as a single atomic unit. Interviewers expect you to discuss chunking, hashing, manifests, and how to ensure correctness at scale.

Hints to consider:

• Generate a manifest upfront listing all chunk IDs, offsets, sizes, and per-chunk cryptographic hashes
• Verify each chunk independently as it arrives; this enables parallelism and quick failure detection
• Store a root hash or signature for the entire file to prevent manifest tampering
• Design for idempotent chunk writes so retries don't corrupt partial state
• Discuss atomic activation: only switch to the new model once the full file is verified

3. Failure Handling and Observability

At this scale, failures are guaranteed. Interviewers want to see how you build resilience and visibility into the system.

Hints to consider:

• Implement heartbeats and timeouts to detect crashed or stalled downloads
• Maintain a central tracker that knows which chunks each peer has; peers can query this to find alternative sources
• Add exponential backoff and retry logic for transient network errors
• Expose metrics (chunks/sec, peers active, bytes transferred) and dashboards so operators can monitor and debug
• Design for graceful degradation: if some servers fail, the rest should still complete successfully

4. Congestion Control and Network Safety

A naive peer-to-peer design can create incast storms and overwhelm NIC buffers. Interviewers test whether you understand practical networking constraints.

Hints to consider:

• Rate-limit each peer's upload and download to avoid saturating individual NICs (e.g., cap at 1Gbps per host)
• Limit the number of simultaneous connections per peer to prevent file descriptor exhaustion
• Use backpressure or token-bucket algorithms to throttle the control plane when the data plane is saturated
• Consider prioritization: production traffic should not be disrupted by model distribution
• Discuss topology awareness: prefer downloading from peers in the same rack to minimize core switch load

Suggested Approach

Step 1: Clarify Requirements

Start by confirming the constraints and success criteria with your interviewer:

• What is the acceptable total distribution time? (This sets the bar for optimization.)
• Are all 1,000 servers identical, or are there multiple datacenters, availability zones, or rack topologies?
• Can we assume servers have sufficient disk space and can write at line rate?
• Is the external repository reliable, or do we need to cache/stage the model inside the datacenter first?
• What happens if a small number of servers fail? Should the system block, retry indefinitely, or report partial success?
• Are there network policies, firewalls, or bandwidth reservations we need to respect?