Definition
The KV cache is the data structure that makes autoregressive LLM inference practical: it stores the attention key and value projections of all previous tokens, so generating token N costs one token's worth of new computation instead of recomputing projections for the entire sequence. Without it, generation cost would grow quadratically as the output lengthens.
The price is GPU memory — the cache grows linearly with sequence length and batch size, and at scale it competes directly with model weights for VRAM. This is why KV cache management dominates LLM serving design: paged attention (allocating the cache in blocks, like virtual memory) reduced fragmentation enough to become the core innovation of vLLM, and prefix caching reuses the KV state of shared prompt prefixes across requests.
How it comes up in interviews
The gateway question of LLM infrastructure interviews: "what does the KV cache store and why?" separates candidates who have served models from those who have only called APIs. Follow-ups go to memory math (cache size per token per layer) and eviction/sharing strategies.