embedding
Every token has its own long list of numbers, which the model learned while it was being trained. When the model reads text, it never sees the letters - only each token's list of numbers. You can think of the list as the word's profile, written in numbers instead of words.
Each token's ID number picks one row from a big lookup table called the embedding table. In Qwen3.5-4B a row is 2,560 numbers. That row, the token's embedding, is what the model actually reads in place of the word. The numbers were learned in training and are fixed afterwards: every time space, “rain” appears, the model reads the same row.
The embedding matrix has one row per vocabulary entry, 248,320 × 2,560 in Qwen3.5-4B. Looking up token id i selects row i, which is the same as multiplying a one-hot vector by the matrix. The rows are parameters, trained with the rest of the model, and each becomes the first state of the residual stream at its position. Where the "profile" picture breaks: a single number in the row rarely means anything on its own, such as "wet" or "plural". What a row says is carried by its direction across many numbers at once.
, where is the 248,320 × 2,560 table: 635,699,200 parameters, 15.1% of the language model's 4,205,751,296. The table is stored at 4 bits in groups of 64 and dequantised on lookup to bfloat16. For space, “rain” (row 10849), values run from −0.043 to 0.049 (quantised in steps of ~0.0025 in its first group of 64), and the row's Euclidean length is 0.62 against a vocabulary median of 0.65. The same matrix scores the output (tied weights), so each row is shaped both by how its token is read and by the hidden states that predict it. The 243 padding rows are all but identical and never score near the top (logits).
Words that are used in similar ways end up with similar lists of numbers. space, “rain”, space, “Rain”, and “雨” (Chinese characters) (the Chinese word for rain) end up with number lists that look very similar, even though nobody ever told the model they mean the same thing.
Each embedding is a point in a space with 2,560 directions: the model's latent space. Training puts tokens that are used alike in rows that point alike. Cosine similarity measures how closely two rows point in the same direction: 1 means identical, 0 means unrelated, and −1 means opposite. space, “rain”'s nearest rows are space, “Rain” (0.75), “Rain” (0.68) and the Chinese “雨” (Chinese characters) (0.62), while a typical row scores about 0.04.
Cosine similarity, , compares directions and ignores length. In the embedding matrix it reflects how tokens are used. Case and spacing variants and translations cluster: space, “pluie” and space, “lluvia” (both 0.38) are #17 and #19 of space, “rain”'s neighbours. A related but differently used word sits further out: space, “snow” is 0.25, far above a typical row but below every form of "rain". Two-dimensional maps of these neighbourhoods (t-SNE, UMAP) distort distances, so the lab shows ranked lists instead.
Cosine similarity runs from 1 (the same direction) through 0 (orthogonal, as unrelated directions nearly are in 2,560 dimensions) to −1 (opposite). Over the table's rows of 2,560 numbers each, space, “rain”'s cosine neighbours are space, “Rain” 0.75, “Rain” 0.68, “雨” (Chinese characters) 0.62 and “rain” 0.61. Further down, space, “pluie” (#17) and space, “lluvia” (#19) are both 0.38, and space, “snow” (#48) is 0.25. The median row scores 0.04, and the 99.9th percentile 0.13, so the rain cluster stands far out of the bulk. That median is signed, and above the 0 around which unrelated directions in 2,560 dimensions scatter (within about ±0.02): space, “rain”'s row leans slightly toward most of the table, a sign of the anisotropy common in embedding tables. These figures are cosines of the dequantised 4-bit rows (lesson 2's probe, 2026-10-15). Input-embedding neighbourhoods are a weak proxy for meaning: they record co-occurrence statistics and, under tied weights, the output role as well. A direction found here need not be the one the model uses deeper in the residual stream, where interpretability work usually looks.