This inference pipeline is slow. Explain how you would debug it.

Problem Statement

Your team operates a production machine learning inference service that processes customer requests in real-time. Over the past week, users have reported increased latency, with P95 response times jumping from 200ms to 800ms. The service handles model inference for a computer vision application that classifies images uploaded by users. The system processes approximately 10,000 requests per minute during peak hours across a cluster of GPU-enabled inference servers.

As the infrastructure engineer on call, you need to systematically diagnose the performance degradation and propose solutions to restore acceptable latency levels. The inference pipeline consists of multiple stages: request ingestion, preprocessing, model inference, postprocessing, and response delivery. No recent deployments have been made to the model or application code, though traffic has grown by approximately 15% over the last month.

Key Requirements

Functional

• Request Processing -- Accept image uploads, perform inference, return classification results
• Multi-stage Pipeline -- Handle preprocessing, GPU inference, and postprocessing stages
• Result Accuracy -- Maintain inference quality while optimizing performance
• Monitoring -- Provide visibility into each pipeline stage for debugging

Non-Functional

• Latency -- Restore P95 latency to under 250ms, P99 under 400ms
• Throughput -- Support 10,000+ requests per minute with room for 2x growth
• Reliability -- Maintain 99.9% success rate during optimization efforts
• Cost Efficiency -- Optimize without dramatically increasing infrastructure costs

What Interviewers Focus On

Based on real interview experiences, these are the areas interviewers probe most deeply:

1. Systematic Debugging Methodology

Interviewers want to see a structured approach to identifying bottlenecks rather than random guessing. They're evaluating whether you understand observability principles and can methodically narrow down root causes in a complex distributed system.

Hints to consider:

• Break down the pipeline into measurable stages and instrument each one
• Use profiling tools to identify whether the bottleneck is CPU, GPU, memory, network, or I/O bound
• Consider both application-level metrics (request queuing, processing time) and system-level metrics (GPU utilization, memory pressure)
• Look for changes in traffic patterns, data characteristics, or resource contention

2. GPU and Hardware Resource Management

For ML infrastructure roles, understanding GPU utilization patterns and bottlenecks is critical. Interviewers look for knowledge of how inference workloads interact with hardware accelerators.

Hints to consider:

• Investigate GPU utilization rates and whether GPUs are idle, saturated, or thrashing
• Consider batch size optimization and whether dynamic batching is being used effectively
• Examine memory transfer overhead between CPU and GPU (PCIe bandwidth)
• Evaluate whether model size or complexity has implicit dependencies on hardware capabilities

3. Batching and Queueing Strategies

Efficient request batching is fundamental to ML inference performance. Interviewers expect candidates to understand the tradeoffs between latency and throughput.

Hints to consider:

• Analyze current batching strategy (fixed vs. dynamic batching) and timeout configurations
• Consider whether micro-batching could balance latency and GPU efficiency
• Evaluate queue depths and whether requests are experiencing head-of-line blocking
• Discuss adaptive batching based on traffic patterns and GPU availability

4. Model Optimization Techniques

While this is an infra role, understanding model-level optimizations shows depth. Interviewers may probe whether you know when to involve ML engineers versus when infra changes suffice.

Hints to consider:

• Consider model quantization (FP32 to FP16 or INT8) to reduce memory and computation
• Evaluate whether model compilation or optimization frameworks (TensorRT, ONNX Runtime) are being used
• Discuss model serving frameworks that provide built-in optimizations
• Determine if model warmup or caching strategies could reduce cold start latency

5. Infrastructure Scaling and Resource Allocation

Interviewers want to see how you think about horizontal vs. vertical scaling and resource allocation strategies for ML workloads.

Hints to consider:

• Analyze whether the cluster has sufficient GPU capacity for current traffic
• Consider autoscaling policies and whether they're responding appropriately to load
• Evaluate CPU-to-GPU ratios and whether preprocessing is becoming a bottleneck
• Discuss pod scheduling, affinity rules, and whether resource requests/limits are properly configured

Suggested Approach

Step 1: Clarify Requirements and Gather Context

Start by understanding the system's current state and recent changes. Ask about the architecture (monolithic service vs. microservices), the specific model being served (size, complexity, framework), and the infrastructure stack (cloud provider, orchestration, GPU types). Confirm the baseline performance metrics before the degradation occurred. Inquire about any recent changes to traffic patterns, data distribution, or infrastructure updates. Establish which metrics are already being collected and what observability tools are available.

Step 2: Instrument and Measure Each Pipeline Stage

Propose adding detailed instrumentation to break down latency by pipeline stage: request queuing time, preprocessing duration, GPU inference time, postprocessing duration, and network transmission time. Set up distributed tracing if not already present to track individual requests through the system. Create dashboards showing P50, P95, and P99 latencies for each stage over time. Monitor GPU-specific metrics including utilization percentage, memory usage, SM (streaming multiprocessor) occupancy, and kernel execution time. Track system-level metrics like CPU usage, memory pressure, network bandwidth, and disk I/O.

Step 3: Identify and Analyze Bottlenecks

Use the instrumentation to pinpoint which stage(s) show degraded performance. For GPU bottlenecks, check if utilization is low (indicating batching issues or overhead) or high (indicating saturation). Examine whether GPU memory is being exhausted, causing swapping or OOM errors. For CPU bottlenecks, profile preprocessing code to identify hot paths. Check for resource contention—are multiple processes competing for the same GPUs? Analyze request queue depths to determine if the system is under-provisioned. Compare current traffic characteristics (image sizes, batch distributions) against baselines to identify anomalies.

Step 4: Propose and Validate Solutions

Based on bottleneck analysis, propose targeted solutions. If GPU utilization is low, implement or tune dynamic batching with appropriate timeout thresholds. If GPUs are saturated, consider horizontal scaling or upgrading to more powerful GPU instances. For memory issues, explore model quantization or reducing batch sizes. If preprocessing is slow, parallelize operations or move them to separate CPU workers. Implement multi-model serving or model replicas if a single model instance is the bottleneck. For queueing issues, add load shedding or priority queues. Discuss how you would A/B test changes in production, monitor for regressions, and gradually roll out optimizations while maintaining reliability.