Next-Token Prediction

every answer
of a language
model is written
one token
at a time.

image: ai-generated (gpt-image-2.5-flare)

a token is a piece of text

split and numbers for illustration

rare words fall apart

split for illustration · try it yourself: tiktokenizer.vercel.app

the vocabulary

262,000

tokens in the vocabulary of Gemma 4 E2B, and everything the model writes is put together from them.

source: gemma 4 model card

in one step, what does the model compute?

the next word
the whole answer
a probability per token
a search on the internet

the field, the barn or the yard?

probabilities for illustration · try it with a real model: alonsosilva-nexttokenprediction.hf.space

the loop

fluent is not the same as true

the model writes the most likely continuation, which is not always the true one.

ask the same question twice: why do you get two different answers?

the model learned
tokens are drawn at random
another model answered
it searched again

temperature

probabilities for illustration, temperature calculated · try it live: poloclub.github.io/transformer-explainer

top k and top p

probabilities for illustration · try it live: poloclub.github.io/transformer-explainer

turn the dial

ask the same question three times at temperature 0.

ask it three times at temperature 1.

switch off the filters: top k = 0, top p = 1, min p = 0.

raise the temperature step by step until the answer falls apart.

in LM Studio, switch to developer mode to see all settings.

everything it sees

how many tokens fit into the context of Gemma 4 E2B?

4,096
32,768
131,072
no limit

why there is a limit

the model was trained on texts up to a certain length.

every new token is related to every token before it.

every token in the context needs memory.

when the context is full, something has to go.

when the context is full

stop at limit

the answer ends when the window is full.

rolling window

the oldest messages are dropped.

truncate middle

the beginning and the latest messages stay.

summarise

the old part is replaced by a short summary.

whatever is dropped no longer exists for the model.

the tractor or the gate?

a. vaswani et al.: attention is all you need, 2017

every token with every token

words instead of tokens and strengths for illustration

Jede Zelle sagt, wie stark ein Wort bei der Berechnung auf ein anderes zurückgreift; wie die Verbindung aussieht, lassen wir offen. Wörter stehen hier vereinfachend für Tokens. Die Zeile „it“ greift am stärksten auf „tractor“ zurück, das ist der Bogen von der Folie davor. Die obere Hälfte bleibt leer, weil ein Token nur auf die Tokens davor schaut; die Matrix ist trotzdem n × n groß. Die Zahl der Zellen wächst mit dem Quadrat der Länge: doppelter Kontext, vierfache Arbeit, und bei 131.072 Tokens sind es 17 Milliarden Zellen, und das in jeder Schicht des Modells. Kubisch wäre es nur, wenn das Programm für jedes neue Token die ganze Matrix neu rechnen würde; das spart es sich mit einem Zwischenspeicher, dem KV-Cache, und genau der braucht den Arbeitsspeicher aus „why there is a limit“.

key takeaway

a language model writes one token at a time, and the whole context flows into the prediction of every single one.

image: ai-generated (gpt-image-2.5-flare)