temperature

Temperature is a dial that controls how adventurous the next-token pick is. Turned low, it sticks closely to the strongest candidate: asked to finish "My favourite colour is", it picks space, “blue”·blue most of the time. Turned up, unlikely choices get more of a chance, and turned too far, it starts picking words that make no sense at all.

A setting that sharpens the odds (below 1) or flattens them (above 1) before a token is sampled; at 1 they are the model's own.

Temperature divides the model's scores before softmax turns them into probabilities. Below 1 it widens the gaps, so the favourite takes more of the chance: at 0.3, "My favourite colour is" comes up space, “blue”·blue 70% of the time. Above 1 it narrows them, so the long tail of unlikely tokens gets more: at 1.5, the dice start landing on tokens that make no sense in the sentence. At 1 the odds are exactly the model's.

Temperature $T$ divides the logits before softmax. Since softmax only sees the differences between scores, $T$ below 1 stretches the gaps, so the favourite takes more of the probability, and $T$ above 1 shrinks them, so the tail gains. In lesson 1, the share of space, “blue”·blue after "My favourite colour is" goes 70%, 38%, 21% and 4.5% at 0.3, 0.7, 1.0 and 1.5.

The tail of a 248,077-token vocabulary is mostly tokens that make no sense in context, so a high temperature does not mainly bring more interesting colours: at 1.5, 40 rolls gave 36 different tokens, among them “INT”INT and “象征” (Chinese characters)象征. Where the dial picture breaks: it changes how the model's odds are used, not the model itself.

$p(T) = \mathrm{softmax}(z / T)$. As $T \to 0$ it tends to the argmax (greedy decoding); as $T \to \infty$, to uniform over all 248,320 rows of the table, the 248,077 tokens and the 243 padding rows alike. At $T$ = 1 the probabilities are the model's own, the ones its training loss scores. In lesson 1's flipbook-colour-qwen, "My favourite colour is" puts space, “blue”·blue at 70%, 38%, 21% and 4.5% at $T$ = 0.3, 0.7, 1.0 and 1.5, and space, “green”·green at 25%, 24%, 15% and 3.6%.

Its 40 seeded rolls per stop (mx.random.categorical(logits / T), seeds 0 to 39) gave space, “blue”·blue 26 times at 0.3, and 36 distinct tokens at 1.5, among them “INT”INT, “#define”#define, “象征” (Chinese characters)象征 and space, “Bukit”·Bukit. Temperature is often combined with top-k or top-p truncation, which removes the tail before drawing (sampling).