sampling

The model only says how likely each next token is; something else has to choose one. Always picking the favourite gives the exact same text every time. Sampling rolls a weighted die instead: a token with a 45% chance comes up about 45% of the time. Roll again, and a different token may come up.

The model only gives odds; a separate step has to pick the winning token. The simplest rule, greedy decoding, always takes the single most probable token, so the same prompt always produces the exact same continuation. Sampling rolls a weighted die instead, picking at random according to each token's probability: a token at 45% comes up about 45% of the time. Run it again and the text can take another turn: in lesson 1, five rolls from "The capital of France is" gave space, “Paris”·Paris only twice, and five different continuations.

Sampling draws the next token from the distribution that softmax gives, often after reshaping it with a temperature. Greedy decoding takes the most probable token: deterministic, and prone to repeating itself over long texts. Sampling gives variety, at the cost of sometimes picking a poor token. Common refinements cut the tail before drawing: top-k keeps the k most probable tokens, and top-p (nucleus sampling) keeps the smallest set whose probabilities add up to p. Where the dice analogy creaks: remember this is a 248,077-sided die whose weights are recalculated from scratch at every single token step.

Given logits $z$ and temperature $T$, a sample is $x \sim \mathrm{Categorical}(\mathrm{softmax}(z / T))$, one of the 248,077 tokens (the 243 padding rows get almost nothing); greedy decoding is the limit $T \to 0$. Truncation renormalises over a subset first: top-k, or nucleus (top-p) sampling (Holtzman et al. 2020, The Curious Case of Neural Text Degeneration), which keeps the smallest set of tokens whose cumulative probability reaches p. The lab's rolls use no truncation. They are drawn in advance by mx.random.categorical(logits / T) keyed by mx.random.key(seed), with the seed recorded, so every roll is reproducible. In lesson 1's France run (flipbook-france-raw-qwen, seeds 0 to 4 at $T$ = 1), space, “Paris”·Paris, at 45%, came up twice in five rolls.