Designing an LLM inference platform is one of the most important system design topics for engineering interviews at AI-era companies like Anthropic, OpenAI, Nvidia, and Google DeepMind. Unlike designing a URL shortener or a Twitter feed, this problem has no widely published reference solution — making it both a differentiator for well-prepared candidates and a trap for those relying on standard resources.
This guide walks through the full system design for an LLM inference platform, structured as it would unfold in a real interview.
What Is LLM Inference and Why Is It Hard?
LLM inference is the process of generating text from a trained large language model given a user prompt. At scale — serving millions of users simultaneously — it is an extraordinarily complex systems problem for several reasons:
- Stateful computation. Unlike a stateless web API, each token generated depends on all previous tokens. This creates memory and state management challenges that do not exist in typical distributed systems.
- GPU memory constraints. Large models require tens or hundreds of gigabytes of GPU VRAM. Serving a 70B parameter model requires careful memory management and often multiple GPUs per request.
- Heterogeneous workloads. A chat prompt of 50 tokens and a document summarisation request with 4,000 tokens have vastly different computational profiles. Efficient batching must handle this heterogeneity.
- Latency-throughput trade-off. Users expect low latency (first token under 200ms), but maximising GPU utilisation requires batching that introduces latency. Balancing these is non-trivial.
Step 1: Clarify Requirements
Before designing anything, a strong candidate asks structured questions:
- Scale. How many concurrent users? Thousands or millions? What is the expected request rate (requests per second)?
- Latency SLA. What is the target for time-to-first-token (TTFT) and time-per-output-token (TPOT)? Is P50 or P99 the relevant metric?
- Model size. Are we serving a single model or multiple? What parameter count? A 7B model and a 175B model require completely different infrastructure.
- Context length. What is the maximum input length? Longer contexts require more KV cache memory.
- Streaming. Do users receive tokens as they are generated, or only when generation is complete?
- Fault tolerance. What is the acceptable downtime? Is this a consumer product or enterprise API?
For this design, assume: 1 million concurrent users, P99 TTFT under 500ms, P99 TPOT under 50ms per token, a 70B parameter model, 8,192 token context length, streaming output, and 99.9% uptime.
High-Level Architecture
The system has five main layers:
- API Gateway — receives user requests, handles authentication, rate limiting, and routing
- Request Scheduler — batches requests, manages queue prioritisation
- Inference Engine Cluster — GPU servers running the model, with KV cache management
- KV Cache Store — manages attention cache across request lifecycle
- Observability Layer — metrics, tracing, alerting
Deep Dive: The KV Cache
The KV cache is the single most important concept to understand for LLM inference system design.
What Is the KV Cache?
In transformer models, the attention mechanism computes Key (K) and Value (V) matrices for each token at each layer. When generating the next token, these matrices from all previous tokens must be recomputed — or cached. The KV cache stores these computed matrices so they do not need to be recomputed on every generation step.
Without KV cache: Generating 1,000 tokens requires 1,000 + 999 + 998 + ... = ~500,000 forward passes. With KV cache: Generating 1,000 tokens requires 1,000 forward passes — a 500x reduction in compute.
KV Cache Memory Requirements
For a 70B parameter model with 8K context:
- Each layer stores K and V matrices of shape (context_length, num_heads, head_dim)
- A 70B model typically has 80 layers and 8 attention heads with dimension 128
- Per token per layer: 2 * 8 * 128 * 2 bytes (fp16) = 4,096 bytes
- Per request at full 8K context: 4,096 * 80 * 8,192 = ~2.5 GB
This means a single H100 GPU (80GB VRAM) can hold roughly 30 concurrent requests at maximum context — a fundamental constraint on throughput.
PagedAttention
The key innovation in production LLM serving (pioneered by the vLLM project) is PagedAttention: managing KV cache in fixed-size pages, similar to virtual memory in operating systems. This eliminates memory fragmentation and allows the system to serve significantly more concurrent requests by sharing cache pages across requests with common prefixes (e.g., system prompts).
Deep Dive: Request Batching
Batching multiple requests together allows the GPU to process them in parallel, dramatically improving throughput. But naive batching has problems:
The Padding Problem
If requests have different input lengths, shorter requests must be padded to match the longest. This wastes GPU computation on padding tokens. For highly heterogeneous workloads (typical in production), padding overhead can waste 30-50% of GPU capacity.
Solution: continuous batching (also called iteration-level scheduling). Rather than batching at the request level (waiting for N requests before processing), the scheduler batches at the token generation level. New requests are inserted into the batch as others complete, keeping GPU utilisation high without waiting.
Priority Queuing
Not all requests are equal. Enterprise API customers may have SLA guarantees. Premium users may have lower-latency tiers. The scheduler must support priority queues while preventing starvation of lower-priority requests.
Deep Dive: Streaming Architecture
Streaming requires the inference engine to push tokens to the user as they are generated, rather than waiting for the full response.
Implementation: Use server-sent events (SSE) or WebSockets at the API layer. The inference engine writes tokens to a per-request queue as they are generated. The API gateway streams these to the client.
Challenge: With millions of concurrent connections, the API gateway must maintain long-lived connections without blocking. This requires an async, event-driven architecture — not a traditional thread-per-request model.
Deep Dive: Multi-GPU and Multi-Node Serving
A 70B parameter model in fp16 requires approximately 140GB of VRAM — more than fits on a single H100 (80GB). Two parallelism strategies are used:
Tensor parallelism: Split individual weight matrices across multiple GPUs. Each GPU holds a slice of each layer. This is low-latency but requires high-bandwidth inter-GPU communication (NVLink).
Pipeline parallelism: Assign consecutive layers to different GPUs. GPU 1 runs layers 1-20, GPU 2 runs layers 21-40, and so on. This has higher throughput but more latency due to pipeline bubbles.
For a 70B model, tensor parallelism across 2-4 H100 GPUs is the typical production approach.
Fault Tolerance and Reliability
GPU failures: GPUs fail more frequently than CPUs. The system must detect inference engine failures quickly (via heartbeat) and redirect in-flight requests to healthy instances. KV cache for interrupted requests is lost — the request must restart from the beginning.
Graceful degradation: Under extreme load, the system should reject requests with a 503 rather than allowing latency to degrade to unacceptable levels. This is preferable to letting all requests time out.
Health checks: Each inference engine exposes a health endpoint. The load balancer performs health checks every few seconds and removes unhealthy nodes from the pool immediately.
Observability
For an LLM serving system, standard HTTP metrics are insufficient. Key metrics to instrument:
- Time-to-first-token (P50, P95, P99) per model and per tier
- GPU memory utilisation per node
- KV cache hit rate (for prefix caching)
- Batch size distribution over time
- Queue depth and wait time per priority tier
- Token generation throughput (tokens/second) per GPU
Common Follow-Up Questions
How would you handle a model that is too large to fit on any combination of available GPUs? Use quantisation (reducing precision from fp16 to int8 or int4) to reduce memory footprint, with a measured trade-off in output quality.
How would you implement prefix caching? Many requests share common prefixes (system prompts, few-shot examples). Cache the KV tensors for these shared prefixes using a trie-based structure, so subsequent requests with the same prefix skip the prefill computation entirely.
How do you handle context windows longer than the model was trained on? Techniques like RoPE scaling, sliding window attention, or retrieval-augmented generation (RAG) can extend effective context. This is an area of active research with real trade-offs.
Frequently Asked Questions
Is LLM inference system design only relevant for AI company interviews? No — as LLMs become embedded in standard products, backend engineers at any large tech company may need to design or integrate with LLM serving infrastructure. But it is most commonly asked at Anthropic, OpenAI, Google DeepMind, and Nvidia.
Do I need to know CUDA to answer this question well? No. Understanding the systems-level constraints (GPU memory, bandwidth, parallelism) is sufficient. CUDA-level knowledge is more relevant for ML infrastructure roles than for system design interviews.
What is the most important concept to understand for LLM serving system design? The KV cache. If you understand what it is, why it exists, and what its memory constraints imply, you can reason about almost everything else in the system.