token
A language model doesn't read letters, or even whole words. It reads text in small pieces called tokens. A piece is often a whole word with the space in front of it, like space, “Paris”, but it can also be part of a longer word, a digit or a full stop. When the model writes, it adds one token at a time.
A token is a chunk of text from the model's fixed vocabulary, which is 248,077 entries in Qwen3.5-4B. It might be a common word with the space in front of it (space, “Paris”), a piece of a longer word (space, “rain” + “drop”), a single digit or a punctuation mark. The model reads text as a sequence of tokens, and writes by adding one token per step (autoregression).
The token is the model's unit of input and output. Tokenization cuts text into entries of a fixed vocabulary of 248,077, and each entry's integer id selects a row of the embedding table. The model produces one probability distribution over the vocabulary at each position, and generation appends one token per forward pass. Lengths, costs and context limits are counted in tokens, not characters or words.
Qwen3.5-4B's vocabulary has 248,077 entries: 248,044 from byte-level BPE (tokenization: the 256 single bytes and the merges built on them) and 33 special tokens, among them the chat template's “<|im_start|>” (special token) and “<|im_end|>” (special token). Ids index a 248,320 × 2,560 embedding matrix; the 243 rows past the vocabulary are padding, which makes the row count a multiple of 256. The 262,144-token context and the per-step cost of autoregressive decoding are both counted in these units, so how a tokenizer cuts a language or a script directly sets what a given text costs.
The space before a word usually travels with the word. We show this space as a faint dot, to make it apparent. So space, “Paris” with its space and “Paris” (without a space) are considered to be different. Likewise “the” is not the same as “The”.
A token usually carries the space in front of it, so spacing and capitals change which token it is: space, “the”, space, “The” and “the” are three different entries, numbers 279, 561 and 1719. The lab draws whitespace as faint ghost marks, as code editors do: · for a space, ¶ for a line break, → for a tab.
The tokenizer attaches a leading space to the word that follows it, so word-initial and word-internal forms are distinct entries, and so are capitalisations: space, “the” 279, space, “The” 561 and “the” 1719 are unrelated ids, each with its own embedding row, learned separately. The lab renders whitespace as ghost marks (·, ¶, →) so that this is visible.
In the byte-level vocabulary a leading space is the byte 0x20, stored as Ġ under GPT-2's byte-to-unicode mapping (Ġrain). Case and spacing variants are distinct ids (space, “the” 279, space, “The” 561, “the” 1719) whose embedding rows are learned independently, yet they end up close: space, “rain”'s nearest row is space, “Rain”, at cosine similarity 0.75 (lesson 2's probe). The lab's ghost marks (·, ¶, →) are drawing conventions only; the bundles store the token text and its raw bytes.
Any text can be written in tokens, even emoji and alphabets the model rarely saw. Any word or symbol not in its vocabulary is spelled out from smaller tokens, down to raw building blocks that can represent any character.
Nothing is unreadable. Text that isn't in the vocabulary is spelled from smaller tokens, down to single bytes, the raw building blocks of digital text. The rain emoji 🌧️ takes four tokens. The lab draws a token that is a lone byte as a box holding its value, such as F0.
Because the vocabulary is byte-level, every UTF-8 string has an encoding, and there is no "unknown" token. The price is that a token need not be a whole character. 🌧 is U+1F327, four bytes F0 9F 8C A7, and it becomes three tokens: the space fused with F0 9F, then 8C, then A7. Its variation selector U+FE0F, which asks for the emoji style, is a fourth token.
Byte-level BPE guarantees coverage: any UTF-8 string tokenizes. 🌧️ with its leading space is four tokens: 10838 (space + F0 9F), 234 (8C), 100 (A7) for the bytes of U+1F327, and 29545 for the variation selector U+FE0F. F0 9F merged because it begins most emoji. Decoding one token in isolation can yield an invalid UTF-8 fragment, and errors="replace" folds an incomplete sequence into a single U+FFFD, losing the byte count. That is why the lab's bundles store each token's raw bytes, and draw one hex box per lone byte.