autoregression

Autoregression literally means predicting a sequence from its own past steps.

A language model can only predict the next token. To write a whole reply, it picks one token, tacks it onto the end, and predicts again from the longer text - over and over, like drawing a flipbook one page at a time. Every page depends on the pages before it, including the ones the model drew itself.

Writing one token at a time, each time adding the token to the text and running the model again on the longer text. A language model predicts only the next token. To write more, it picks one, adds it to the text, and is asked again: its own output becomes its next input. Every reply is built this way.

In lesson 1, "Once upon a time, there was a" grew by space, “young”·young, space, “and”·and, space, “curious”·curious… one token per step, each step with its own odds: space, “and”·and won with only 22%, space, “named”·named with 98%. By itself, nothing makes the loop halt at the end of an answer: it only stops when it picks a special end-of-turn token, or hits a length limit.

The model gives a distribution over the next token given everything before it. Generation chains these steps: choose a token (by sampling or by taking the favourite), append it, and run again. The probability of a whole text is then the product of its step probabilities, each conditioned on the text so far.

A choice can't be taken back: every later step is conditioned on it, which is why one unlikely roll can send a reply somewhere else entirely. Where the flipbook picture breaks: a flipbook's pages are drawn in advance, while each of these is drawn only once the page before it exists.

An autoregressive model factorises a sequence's probability as $p(x_{1:n}) = \prod_t p(x_t \mid x_{<t})$. Generation draws each $x_t$ from the model's conditional and feeds it back. Training scores every position at once against the true previous tokens (teacher forcing), so at generation time the model conditions on its own outputs, a distribution it was never trained on directly (exposure bias); early errors compound.

Generation ends on an end token, in Qwen's chat template “<|im_end|>” (special token)<|im_end|>, or at a length limit; a raw continuation has no natural end. In lesson 1's story run (flipbook-story-qwen), space, “and”·and won at 22% and space, “named”·named at 98%.