Definition
The context window is the maximum number of tokens — prompt plus generated output — an LLM can handle in one request. It is measured in tokens, not words or characters: subword tokenization averages roughly 3–4 characters per English token, which is also why API pricing and limits are token-denominated.
Long contexts are expensive for a structural reason: full self-attention compares every token with every other token, so prefill compute grows quadratically with sequence length — doubling the context roughly quadruples attention cost — while the KV cache grows linearly and must be re-read for every generated token. This scaling is why long-context serving leans on paged and sparse attention and prompt caching, and why "stuff everything into the window" competes with retrieval (RAG) rather than replacing it: attention over a million tokens costs real money and latency on every single request, while retrieval pays once to index and fetches only what is relevant.
How it comes up in interviews
The standard tradeoff question is long-context vs RAG: interviewers want the quadratic-attention reasoning and the per-request vs at-ingest cost comparison, not just feature-sheet numbers. "Why are limits in tokens?" is the warm-up version.