softmax

The model gives every possible next token a score. Softmax is an operation that turns those scores into percentages that add up to 100%. It strongly favours the leaders: a token slightly ahead in score ends up far ahead in chance, so most tokens get almost nothing.

Softmax is a mathematical operation that turns a list of scores into probabilities that add up to 100%. At each step, the model gives every token in its vocabulary a score, where a higher value means it is more likely to come next.

Softmax raises a constant to the power of each score, so a token a little ahead in score ends up far ahead in probability, then divides by the total, so that the probabilities sum to 100%. Most tokens end up with almost nothing; a few make up almost all the chance. After "The capital of France is", space, “Paris”·Paris takes 45%, and the other 248,076 tokens share the remaining 55%.

Softmax maps the logits $z$, one per vocabulary entry, to probabilities $p_i = e^{z_i} / \sum_j e^{z_j}$. Exponentiating makes every probability positive and turns differences in score into ratios of probability: a token 1 point ahead is about 2.7 times as likely, 3 points ahead about 20 times. Only differences matter, so adding the same number to every score changes nothing. The name reflects what it does: it is a smooth version of picking the maximum, and the wider the gaps, the closer it comes to putting all the probability on the top score.

$\mathrm{softmax}(z)_i = \exp(z_i) / \sum_j \exp(z_j)$, over Qwen3.5-4B's 248,320 logits (its 243 padding rows included; they never score near the top). Its outputs are positive and sum to 1 (100%). It is invariant to adding a constant, $\mathrm{softmax}(z + c) = \mathrm{softmax}(z)$, so implementations subtract $\max_i z_i$ first for numerical stability, and a log-probability is $z_i - \log \sum_j e^{z_j}$.

A logit gap of 1 is a probability ratio of $e$, about 2.7; a gap of 3, about 20. Dividing the logits by a temperature $T$ first sharpens ($T < 1$) or flattens ($T > 1$) the distribution, and as $T \to 0$ it tends to the argmax. In lesson 1's France run (flipbook-france-raw-qwen), space, “Paris”·Paris gets 45%, and the other 248,076 tokens (with the padding rows, which get almost nothing) share the remaining 55%.