Definition
Continuous batching — also called in-flight batching — is a scheduling technique for language-model serving in which the batch is re-formed at every decoding step. Static batching groups a fixed set of requests and runs them until all are done, so a batch containing one long generation leaves the finished slots idle. Continuous batching evicts each sequence as it completes and admits a waiting request into the freed slot immediately.
The gain is large because generation lengths vary enormously within a realistic workload, so static batches spend much of their time mostly empty. It is the default in modern serving stacks and pairs naturally with paged attention, which stores KV cache in fixed-size blocks so sequences of different lengths can share GPU memory without fragmentation. The trade-off to acknowledge is fairness and tail latency: an admission policy that always favors throughput can starve individual requests.
How it comes up in interviews
This is the expected answer to "how do you raise GPU utilization on an inference server?" Explain why static batching wastes slots when generation lengths differ, then connect it to memory: the KV cache is the binding constraint on how many sequences can be in flight, which is why paged attention usually comes up in the same breath.