logits

Before the model gives its odds, it gives every possible next token a raw score. A high score means 'likely next'; a low or negative score means 'unlikely'. These raw scores are called logits, and the next step, softmax, turns them into percentages.

Logits are the raw 'scores' from lesson 1, before softmax turns them into odds. At the end of a pass, the model's final output row is lined up against every row of the embedding table to produce one score per token: the closer the match, the higher the logit. They can be any size and either sign, and most are a little below zero. After lesson 2's raindrop sentence, the top score is 18.75, shared by space, “It”·It and space, “What”·What.

Unlike cosine similarity, which only compares direction, this match score also grows with the length of the rows - a calculation lesson 9 will open up.

The logits are the model's final output vector, after a last normalisation, scored against every row of the output table: one dot product per entry, 248,320 in all, the 243 spare rows included. With tied weights, that table is the embedding table itself. A logit grows with how closely the output vector points along a row and with how long both are, which is why the lab speaks of "lining up" rather than of cosine similarity. Only differences between logits matter to softmax, so their zero means nothing in itself: "most are a little below zero" describes this model's scale, not a rule.

$z = E \, \mathrm{RMSNorm}(h)$, where $h$ is the last layer's output at the final position and $E$ the tied 248,320 × 2,560 table, so $z_i = \lVert e_i \rVert \, \lVert \hat h \rVert \cos \theta_i$. After lesson 2's sentence ("A single raindrop fell from a height of 1000 meters.", bundle dictionary-logits-qwen) the 248,320 logits have a median of −0.9 and run up to 18.75; 69% are negative, and 97% lie between −5 and +5. The top two, space, “It”·It and space, “What”·What, tie at 18.75, consistent with bfloat16's spacing of 0.125 between 16 and 32, and take 13% each after softmax. The 243 padding rows all score within 0.02 of −2.78. Logits are defined only up to an added constant, so log-probabilities, $z_i - \log \sum_j e^{z_j}$, are what compares across contexts. The name is borrowed from the logit, the log-odds $\log(p / (1 - p))$, which Joseph Berkson named in 1944 from "logistic unit", after the probit: with two classes, the difference between their two logits is exactly the log-odds. Lesson 1 shows these scores only as percentages; lesson 9 names the dot product.