tied weights

The model has one big table that turns each token into a list of numbers on the way in. On the way out, it has to turn its final numbers back into a score for every possible next token. With tied weights, it uses the very same table for both jobs instead of keeping a second one.

A model needs a table to turn tokens into rows of numbers on the way in (the embedding table) and a table to score its output against every token on the way out. With tied weights, both jobs share the exact same table instead of using two separate ones. Qwen3.5-4B ties them: its 248,320 × 2,560 table - 635,699,200 numbers, or 15% of the whole model - is stored just once. When we recompute the output scores directly from the input table, they match the model's own scores to within rounding. (Many larger models, where a second table is a smaller fraction of the total size, keep two separate tables instead.)

The input embedding maps a token id to its row; the output layer scores the final hidden state against the rows of a matrix of the same shape (logits). Tying makes them one matrix, so each row is shaped by two jobs: how its token is read, and which hidden states should predict it. It saves a second table's parameters, and it can make the model better too (Press and Wolf, 2017). Many larger models keep two tables, where a second one costs a smaller share of the whole.

Qwen3.5-4B's config sets tie_word_embeddings to true. Its table is 248,320 × 2,560 = 635,699,200 parameters, 15.1% (about 15%) of the language model's 4,205,751,296; untied, the model would carry 4.84 billion. We checked the tie rather than trusting the config: lesson 2's probe recomputed all 248,320 logits as the final output vector scored against the input table, and they matched the model's own to within 0.06, the size of its rounding steps. Weight tying for language models was proposed by Press and Wolf (2017, Using the Output Embedding to Improve Language Models) and Inan et al. (2017). For interpretability it means that input-side neighbourhoods and output scores share one geometry (embedding), and the logit lens reads the stream with the very matrix that wrote its first state.