Daily Tech Interview Quiz

Not quite.

Distillation uses a large teacher model to generate outputs (or richer signals like token distributions) that a smaller student model is trained to match, transferring much of the capability at a fraction of the serving cost. It is a primary lever behind fast, cheap production models and behind serving tiers — you pay the teacher's cost once at training time instead of on every inference request.

Not quite.

In MoE layers, a learned router sends each token to a few experts (say 2 of 8), so per-token compute involves only the active experts' parameters while total capacity is the sum of all of them. The catch is memory and serving: all experts must be resident, and batching gets trickier because different tokens activate different experts. 'Total vs active parameters' is the key vocabulary interviewers listen for.

Not quite.

No weights change at inference time — the model conditions on the examples in its context and continues the demonstrated pattern, a capability that emerges from pretraining on vast text with recurring formats. This 'few-shot' behavior is the practical basis of prompt engineering: examples pin down output format and style more reliably than instructions alone, at the cost of tokens on every request (which prefix caching can amortize).

Not quite.

The sequential loops contribute O(n) + O(n) and the nested loop contributes O(n²); the total is O(2n + n²), which simplifies to O(n²) because Big-O keeps only the dominant term and drops constant factors. Writing O(2n + n²) as a final answer is a classic sign of not simplifying.