LLM & AI Engineering Quiz

During autoregressive generation, every new token attends to all previous tokens. Without caching, generating token N would recompute the key and value projections for all N-1 earlier tokens. The KV cache stores those projections once, making each decode step's cost proportional to one token instead of the whole sequence — at the price of significant GPU memory, which is why KV cache management dominates LLM serving design.

Prefill processes every prompt token in one parallel pass — large matrix multiplications that saturate GPU compute. Decode then emits one token per step, where the bottleneck is streaming weights and KV cache from GPU memory, not arithmetic. Schedulers like continuous batching exist largely to keep decode steps from wasting the GPU while balancing latency between the two phases.

Temperature scales the logits before sampling: high temperature flattens the distribution (more randomness), low temperature sharpens it, and zero collapses it to always picking the argmax token. Greedy decoding is mostly deterministic but not creative; note that batching nondeterminism and floating-point ordering can still cause tiny run-to-run variations in practice.

Top-p sorts tokens by probability and keeps the smallest prefix whose probabilities sum past the threshold p (e.g., 0.9), then renormalizes and samples from that nucleus. Unlike top-k's fixed count, the candidate set adapts: confident distributions yield a tiny nucleus, uncertain ones a large one. That adaptivity is why top-p is the more common default.

Embedding models are trained so that texts with similar meaning land near each other (by cosine similarity or dot product), even with zero word overlap — 'How do I reset my password?' sits near 'account recovery steps'. That's what lets a query find relevant documents semantically. Embeddings are lossy and not reversible, and individual dimensions have no human-readable meaning.

RAG retrieves relevant documents (usually via embedding search) and injects them into the prompt, so answers can be grounded in current or private data without retraining the model. Grounding also gives the model something concrete to cite, which cuts hallucination. It changes no weights — that distinction from fine-tuning is a favorite interview probe.

INT4 stores each weight in 4 bits instead of 16, shrinking the model roughly 4x — often the difference between needing four GPUs and one. Because decode is memory-bandwidth-bound, moving fewer bytes also speeds it up. The cost is precision: aggressive quantization can measurably degrade quality, so teams evaluate quantized models before shipping them.

Full self-attention compares every token with every other token: O(n²) compute in sequence length n during prefill, plus a KV cache that grows linearly and must be re-read for every generated token. Doubling context roughly quadruples prefill attention cost. This scaling is why long-context serving leans on techniques like paged/sparse attention and prompt caching.

With static batching, a batch is fixed at start: short generations finish early and their GPU slots idle until the longest one completes, and new arrivals queue. Continuous batching operates per decode step — finished sequences leave, queued requests join immediately — keeping the GPU full. It's the single biggest throughput win in modern serving engines like vLLM.

A cheap draft model guesses the next few tokens; the large model then scores all of them in a single forward pass — verification in parallel is much cheaper than generating one token at a time. Accepted tokens are kept, and the first rejection point falls back to the large model's choice, so output quality is provably unchanged. Speedup depends on the draft model's acceptance rate.

The model's weights don't contain private internal docs, so unaided answers are guesses. Retrieving the relevant passages and constraining the model to answer from them — with an explicit 'say you don't know if it's not in the context' instruction — attacks the root cause. Higher temperature makes hallucination worse, and batch size or output length are unrelated.

LoRA keeps all base weights frozen and learns a pair of small low-rank matrices per adapted layer — typically well under 1% of the model's parameters — slashing GPU memory for gradients and optimizer state. The adapters can be merged into the weights or hot-swapped at serving time, which is also how one base model cheaply serves many fine-tuned variants.

A whole-word vocabulary would need millions of entries and still fail on new words; characters keep the vocabulary tiny but make sequences (and attention cost) enormous. Subword tokenization sits in between: a vocabulary of tens of thousands of frequent fragments that can compose any string, including code and rare words. This is also why token counts, not word counts, drive context limits and API pricing — English averages roughly 3-4 characters per token.

Self-attention computes a score between every pair of token positions, so prefill cost grows quadratically with prompt length — doubling the context roughly quadruples the attention compute. This is the root reason long contexts are expensive, why long-context support is an engineering achievement, and why techniques like sliding-window or sparse attention exist. During decode, each new token attends to all previous ones, which the KV cache makes O(n) per token instead of recomputing O(n²).

Attention is a weighted sum over a set of tokens — permute the input and you get the same result, so word order would be invisible. Positional encodings inject order information; rotary embeddings (RoPE) do it by rotating query/key vectors by position-dependent angles, which encodes relative distances directly in the attention dot product and extrapolates better to longer sequences. This 'attention is a set operation' insight is a favorite interview probe.

Prompting requires no training run, ships instantly, rolls back by editing text, and works across model upgrades — so it is the default; modern models follow instructions and few-shot examples remarkably well (in-context learning). Fine-tuning earns its cost when the behavior must be deeply consistent, the prompt would need to be huge on every call, or you need a domain style/format the model cannot hold in context. 'Start with prompting, fine-tune only when prompting demonstrably fails' is the expected answer shape.

Bi-encoder retrieval embeds the query and documents independently, which is fast but loses fine-grained interaction between their words. A cross-encoder reranker reads the query and each candidate together through one model, capturing exact-term and reasoning-level matches — far more accurate, far too slow to run on the whole corpus. So the pattern is retrieve top-100 cheaply, rerank to top-5 accurately: a classic recall-then-precision funnel.

Distillation uses a large teacher model to generate outputs (or richer signals like token distributions) that a smaller student model is trained to match, transferring much of the capability at a fraction of the serving cost. It is a primary lever behind fast, cheap production models and behind serving tiers — you pay the teacher's cost once at training time instead of on every inference request.

In MoE layers, a learned router sends each token to a few experts (say 2 of 8), so per-token compute involves only the active experts' parameters while total capacity is the sum of all of them. The catch is memory and serving: all experts must be resident, and batching gets trickier because different tokens activate different experts. 'Total vs active parameters' is the key vocabulary interviewers listen for.

No weights change at inference time — the model conditions on the examples in its context and continues the demonstrated pattern, a capability that emerges from pretraining on vast text with recurring formats. This 'few-shot' behavior is the practical basis of prompt engineering: examples pin down output format and style more reliably than instructions alone, at the cost of tokens on every request (which prefix caching can amortize).