Definition
Self-attention is the mechanism by which each position in a sequence builds its representation by attending to all other positions. Every token is projected into a query, a key, and a value; the similarity between one token's query and every token's key produces attention weights, and the output is the weighted sum of the values. Multi-head attention runs several of these in parallel with different projections so distinct relationships can be captured at once.
The cost is the part that shapes system design: comparing every token to every other token is quadratic in sequence length, in both computation and the attention matrix's memory. This is the root cause of long-context expense and the reason for a large body of work on cheaper variants — FlashAttention reorganizes the computation to avoid materializing the full matrix, while grouped-query and multi-query attention shrink the KV cache by sharing key and value heads.
How it comes up in interviews
Expect to be asked why long contexts cost so much more than short ones; the quadratic term is the answer. In coding rounds, implementing scaled dot-product attention from scratch — including the scaling factor and the causal mask — is a common exercise, and forgetting the mask in a decoder is the classic mistake.