Definition
The transformer is a neural network architecture that processes a whole sequence in parallel using attention, rather than stepping through it one token at a time as recurrent networks did. A decoder-only language model — the shape used by most current chat and completion models — is a stack of identical blocks, each containing masked self-attention followed by a feed-forward network, with residual connections and normalization around both.
Its decisive property is parallelism during training: every position in a sequence can be computed at once, which is what made training on internet-scale corpora practical. Because attention itself carries no notion of order, position must be injected separately through positional encodings. Inference has the opposite character from training — tokens are generated one at a time, each attending to everything before it, which is exactly why the KV cache exists and why serving cost is dominated by memory bandwidth rather than raw compute.
How it comes up in interviews
For AI engineering roles you are rarely asked to derive the architecture, but you are expected to reason about its consequences: why context length is expensive, why the KV cache dominates serving memory, and why prefill and decode have such different performance profiles. Connecting architecture to serving economics is the differentiator.