Definition
Tokenization is the process of converting raw text into the discrete units a model consumes. Modern language models use sub-word schemes such as byte pair encoding, which start from characters and iteratively merge the most frequent adjacent pairs into a fixed-size vocabulary. Common words become single tokens, rare words split into pieces, and any input remains representable rather than being mapped to an unknown token.
Everything downstream is denominated in tokens: context windows, per-request pricing, throughput limits, and latency all scale with token count rather than characters. The mapping is not uniform across languages or content types — code, non-Latin scripts, and unusual formatting can consume noticeably more tokens per unit of meaning, which turns tokenization into a real cost consideration when serving a global user base.
How it comes up in interviews
This appears in two places. In coding rounds, implementing a byte pair encoding trainer or encoder is a self-contained problem with a clean data-structure core. In design rounds, it shows up in capacity math — estimating tokens per request is the first step of any serving-cost or throughput estimate, so an interviewer asking "how much will this cost to run?" is asking a tokenization question.