prefilling and decoding

Before a model writes anything, it reads your whole message in one go. A model works at two speeds. The first, where everything is read in at once, is called prefilling. The second, called decoding, is where it writes its reply one token at a time: each token takes another full run of the machine, which is why replies stream back bit by bit.

The two speeds of running a model: prefilling reads the whole prompt in one pass; decoding then writes one token per pass. Before the model writes anything, it reads the prompt, all of it at once, in one pass: prefilling. That pass already gives the odds for the first new token. After that, each new token takes a pass of its own: decoding.

In lesson 1, the 8 tokens of "Once upon a time, there was a" go in together, and ten new tokens take ten passes, one prefill pass and nine decode passes. That is why a long reply streams out token by token. How the model remembers the earlier tokens without re-reading the whole text at every decode step is covered in Lesson 7.

Prefill (prefilling) runs the whole prompt through the model in one forward pass, every position in parallel, and its last position already gives the odds for the first new token. Decode (decoding) then runs one forward pass per new token: the new token is the only new input, and what the model worked out for earlier positions is kept rather than recomputed (Lesson 7). So a reply of 10 tokens after a prompt of 8 takes 1 prefill and 9 decodes. The two feel different: prefill sets the wait before the first token, decode the pace at which the rest streams out.

Prefill processes the prompt's tokens as one batch: matrix–matrix products, which keep the hardware's arithmetic busy, filling each layer's memory of the context. Each decode step processes one token: for a single reply, matrix–vector products that read every weight once per token, so on most hardware decode is bound by memory bandwidth rather than arithmetic.

For Qwen3.5-4B at 4 bits that is about 2.37 GB of weights read per token decoded. Its 8 full-attention layers keep a KV cache that grows with the text, and its 24 linear-attention layers a state of fixed size (Lesson 7). In lesson 1's story run (flipbook-story-qwen), an 8-token prompt and 10 new tokens are 1 prefill and 9 decodes, because the prefill's last position already yields the first new token's distribution. "Time to first token" measures prefill; "tokens per second", decode.