Definition
Time to first token is the interval between a request arriving and the first output token being emitted. It is dominated by prefill: the model must process the entire prompt before it can generate anything, so TTFT grows with input length, and it is compute-bound in a way that per-token generation is not. Once generation starts, the relevant metric becomes inter-token latency, which is bound by memory bandwidth instead.
Splitting serving latency into these two phases is what makes streaming interfaces worth building and what makes serving trade-offs legible. Batching improves total throughput but can delay an individual request's prefill; a long system prompt raises TTFT for every request; prefix caching lets a shared prompt skip prefill work it has already done. Optimizing the wrong phase is a common and expensive mistake.
How it comes up in interviews
For LLM serving questions, defining latency targets separately for first token and subsequent tokens immediately raises the level of the conversation — a chat interface needs fast TTFT and merely adequate throughput, while a batch summarization job is the reverse. Naming prefill versus decode, and their different hardware bottlenecks, is the detail interviewers are listening for.