Multi-Tenant CI/CD Workflow System

Reference Answer

For a full example answer with detailed architecture diagrams and deep dives, see our Design CI/CD System guide.

Problem Statement

Design a scalable, fault-tolerant CI/CD system for a multi-tenant environment that schedules and executes user-defined workflows in response to git pushes. The system must handle workflow execution, job scheduling, real-time status updates, and ensure exactly-once execution semantics.

The system receives push events via API calls from an internal service containing the repository ID and commit hash. Workflows are a sequence of jobs defined within a single YAML file in a static location for each repository. Users should be able to view the output and status of jobs as they are running.

Key Requirements

Functional

• Trigger workflows on each git push using repository-defined YAML config
• Schedule and execute a linear sequence of jobs per workflow (not a DAG)
• Multi-tenant isolation -- tenant-level resource allocation and fairness
• Real-time status and logs -- users can view job output as it runs
• Exactly-once execution -- each job must execute exactly once, even under failures

Non-Functional

• Fault tolerance -- survive worker crashes, scheduler restarts, and network partitions
• Horizontal scalability -- all components should be stateless where possible
• Low latency -- minimize time from git push to first job starting
• Consistency -- job state must be durable and consistent across components

What Interviewers Focus On

Based on real interview experiences, these are the areas interviewers probe most deeply:

1. Exactly-Once Execution (Most Emphasized)

This is the single most tested topic. Interviewers want to see how you prevent duplicate job runs under failure scenarios -- worker crashes mid-execution, scheduler restarts, or network partitions.

Hints to consider:

• Idempotency tokens per job + commit hash
• Database as source of truth for job state with atomic transitions
• Distributed locking or optimistic concurrency control
• Fencing tokens to prevent stale workers from committing results

2. Stateless Scheduler Design

Interviewers value architectures where the scheduler holds no in-memory state, enabling effortless horizontal scaling.

Hints to consider:

• Persist all step entries in database upfront (all PENDING status)
• Only enqueue the first step to a message queue
• Workers update step status in DB after completion
• Use CDC (Change Data Capture) to notify the scheduler of state changes
• Scheduler queries DB for the next pending step and enqueues it
• This creates a fully stateless scheduler that can scale horizontally

3. Multi-Tenant Isolation

How do you prevent noisy neighbors and ensure fair resource allocation across tenants?

Hints to consider:

• Per-tenant job queues or weighted fair scheduling
• Resource quotas and rate limiting per tenant
• Containerized execution (K8s pods / Docker) for isolation
• Priority tiers for different tenant SLAs

4. Real-Time Status and Log Streaming

Interviewers sometimes extend the problem to include a front-end UI that shows live workflow progress.

Hints to consider:

• WebSocket or SSE connections for live updates
• Log aggregation pipeline (workers stream to a log service)
• Polling fallback for reliability
• Job status state machine: PENDING -> RUNNING -> SUCCESS / FAILED

5. Execution Infrastructure

Kubernetes and Docker are often explicitly mentioned as expected technologies.

Hints to consider:

• K8s Jobs or pods for isolated execution per step
• Resource requests/limits per container for tenant fairness
• Ephemeral runners that spin up and tear down per job
• Secrets injection scoped per repository/tenant

Suggested Approach

Step 1: Clarify Requirements

Confirm the workflow model (linear vs DAG), scale expectations, and latency requirements. Most interviews scope this to linear sequential jobs.

Step 2: High-Level Architecture

Sketch these core components:

• Webhook Receiver -- accepts git push events, deduplicates, creates workflow records
• Workflow Scheduler -- stateless service that creates job entries and manages step progression
• Message Queue -- decouples scheduling from execution
• Worker Pool -- executes jobs in isolated containers
• Database -- source of truth for workflow/job state
• Status Service -- serves real-time updates to the UI

Step 3: Deep Dive on Exactly-Once

This is where you should spend the most time. Walk through failure scenarios:

• What happens if a worker crashes mid-job?
• What if the scheduler crashes after creating jobs but before enqueueing?
• How do you detect and recover from stuck jobs?

Step 4: Address Multi-Tenancy

Discuss tenant isolation, fair scheduling, and resource limits. Show you can prevent one tenant from starving others.

Real Interview Quotes

"The interviewer seemed to focus heavily on exactly-once execution. They kept emphasizing to just treat each job as a single task, not even requiring it to be linear. They also asked about how to design for multi-tenant scenarios."

"One approach is to have a DB entry for each step. Your scheduler creates all step entries (all in pending status), but only puts the first step into a queue for workers to run. Workers update the step entry status. After the status changes, you use CDC to notify your scheduler. This way you achieve a completely stateless scheduler."

"Main focus was on how to implement exactly-once execution under the constraints of being fault-tolerant and scalable."

"Design a CI/CD job scheduler that utilizes tech such as K8s and Docker. Discuss how you would approach the design and the key components involved."