How LLMs Actually Work: The Next-Token Loop

You have typed a prompt into ChatGPT and watched a paragraph appear word by word. It feels like the model is thinking, planning a sentence, then writing it out. It isn't. Under the fluent output is a loop so simple you can run it by hand: score every possible next token, turn those scores into probabilities, pick one, glue it onto the end of the text, and repeat.

That's the whole mechanism. No plan, no lookahead, no sentence-level intent — just a next-token guess, conditioned on everything written so far, made a few hundred times in a row. A 400-billion-parameter model and a toy lookup table run the same loop; scale is only what makes each individual guess uncannily good.

This walks through every stage of that loop with real numbers: logits, temperature, softmax, top-k, top-p, and the final weighted draw. When you finish, you can step through the exact same loop live on YeetCode — drag the knobs and watch every number recompute.

What is a logit?#

The model's final layer emits one raw score — a logit — for every token in its vocabulary. A logit can be any real number. It has no upper or lower bound, and a column of them doesn't sum to anything meaningful. Bigger just means "the model likes this token more here."

Take the prompt The cat sat on the. A tiny model that only knows a handful of continuations might score four candidates like this:

mat scores highest, so the model prefers it — but 3.4 isn't a probability. It isn't 340% or 3.4-out-of-anything. To sample a token you need a real distribution: non-negative numbers that add up to 1. Converting logits into that distribution is the job of exactly one function.

How does softmax turn scores into probabilities?#

Softmax is the function that maps any list of real numbers to a valid probability distribution. Two operations do all the work: exponentiate each score to force it positive, then divide by the total so everything sums to 1.

Ignore the T for a moment (it's 1 by default). Here is softmax on the four logits above:

Now they mean something: mat 68.7%, sofa 18.7%, roof 8.4%, floor 4.2% — and they sum to 1.0. (In real code you subtract the max logit before exponentiating so exp() never overflows; because that constant cancels in the division, the probabilities come out identical.)

The name is precise: softmax is a soft argmax. Hard argmax would hand 100% to mat and 0% to everything else. Softmax instead splits a fixed budget of 1.0 across all candidates. Raise one logit and its share grows only by stealing probability from the others — there's no free mass.

Temperature: the one knob that changes everything#

Temperature T is a single number you divide the logits by before softmax. It's the dial between "safe and repetitive" and "creative and risky," and it works by stretching or squashing the gaps between scores.

• T < 1 magnifies the gaps. Dividing 3.4 and 2.1 by 0.5 gives 6.8 and 4.2 — a wider gap — so after softmax the top token grabs even more mass. The distribution sharpens toward the winner.
• T > 1 shrinks the gaps. Dividing by 2 gives 1.7 and 1.05, closer together, so probability flattens toward uniform and long-shot tokens get a real chance.
• T = 1 is a no-op.

You can measure how spread out a distribution is with Shannon entropy, in bits:

For our four-token guess, H ≈ 1.32 bits. The maximum for four options is log₂(4) = 2 bits (a perfectly uniform 25/25/25/25 split), and the minimum is 0 bits (all mass on one token). Entropy is the honest "how undecided is the model" meter: crank temperature up and entropy climbs toward 2; crank it down toward 0.1 and entropy collapses toward 0 as the top token swallows everything. The softmax playground in the deep dive recomputes this entropy bar live as you drag the temperature slider.

Top-k and top-p: trimming the tail before you sample#

Sampling straight from the softmax distribution has a flaw: even a token with 0.5% probability can occasionally get drawn, and one bizarre word can derail a whole paragraph. Two filters cut the tail before the draw.

With our distribution [0.687, 0.187, 0.084, 0.042], top-k with k=2 keeps mat and sofa, drops roof and floor, and renormalizes: 0.687/0.874 = 78.6% and 0.187/0.874 = 21.4%. Top-p with p=0.9 keeps tokens until the running total crosses 0.9 — mat (0.687) then sofa (0.874) then roof (0.958 ≥ 0.9) — so three tokens survive. The key difference: top-k always keeps the same number of tokens; top-p keeps fewer when the model is sure and more when it's hesitant, which is why nucleus sampling adapts where top-k is rigid.

Both filters renormalize after masking so the survivors still sum to 1.0 — you can't sample from a distribution that doesn't add up.

Walkthrough: generating from "The cat sat on the"#

Here is the full loop on the toy model, using greedy decoding (always take the argmax). Greedy ignores temperature because the ranking never changes, so we can read the winner straight off the logits. Each row is one trip through the loop; the chosen token gets appended and fed back as the new context.

Four tokens later the running text reads: "The cat sat on the mat and purred softly." Notice what actually happened. At no point did the model decide to write a sentence about a cat. It only ever answered one question — what comes next given everything so far? — and the coherent sentence emerged from four independent next-token guesses, each conditioned on the growing context. That feedback (predict one, append, re-read) is autoregression, and it is the entirety of generation.

Swap greedy for sampling and step 3 might land on yawned instead of purred; raise the temperature and stared gets a real shot. Same loop, different roll of a weighted die.

What to remember#

• An LLM emits one logit per vocabulary token — raw, unbounded scores, not probabilities.
• Softmax (exp then normalize) turns those logits into a distribution that sums to 1.
• Temperature divides the logits before softmax: < 1 sharpens, > 1 flattens. Entropy measures the spread in bits.
• Top-k keeps a fixed number of candidates; top-p keeps an adaptive nucleus by probability mass. Both renormalize.
• You then sample (or take the argmax), append the token, and repeat. That loop is the whole model.

Keep going#

You now know the mechanism, but two things were black boxes here: where the logits come from (a learned network, not a lookup table) and how the model reads context (attention over embeddings). Follow the thread:

• From Hand-Written Rules to LLMs: How AI Learned to Learn — why prediction beat hand-coded logic.
• Neural Networks from Scratch: How Machines Actually Learn — the machinery that produces the logits.
• Tensors and PyTorch: The Data Structure That Runs Deep Learning — the arrays every one of these numbers lives in.
• Transformers, Part 1: Tokenization, Embeddings & Positional Encoding — how text becomes something the model can score.

Then open the interactive deep dive to type your own prompt and step the real generation loop — logits → temperature → softmax → top-k → top-p → sample → append — one token at a time, with every number computed live.

FAQ#

Does an LLM plan a whole sentence before writing it?#

No. The model predicts exactly one token at a time, conditioned on all the text so far, and has no representation of the sentence it's building toward. Coherence is emergent: each next-token guess is good enough, given the growing context, that the tokens chain into fluent prose. That's why models can paint themselves into a corner mid-sentence — there was never a plan to abandon, only the next guess.

What's the difference between top-k and top-p sampling?#

Top-k keeps a fixed number of candidates — the k most probable tokens — no matter how the distribution is shaped. Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities sum to at least p, so the count changes with the model's confidence: a handful when it's sure, many when it's uncertain. Top-p adapts to the distribution; top-k does not. Many systems apply both, top-k first then top-p, and renormalize after each.

What does temperature actually do to the output?#

Temperature divides every logit before softmax, which rescales the gaps between scores. Below 1 it widens the gaps so the top token dominates — output becomes focused and repetitive, approaching greedy decoding as T → 0. Above 1 it narrows the gaps so probability spreads toward the tail — output becomes diverse and occasionally strange. You can watch this directly as Shannon entropy: low temperature drives entropy toward 0 bits, high temperature toward its maximum of log₂(vocabulary size).

Why not just always pick the highest-probability token?#

Always taking the argmax is greedy decoding, and it's perfectly valid — deterministic and reproducible. But it's often dull and repetitive because it ignores the rest of the distribution entirely, and it can get stuck in loops ("the the the"). Sampling from the trimmed distribution keeps output varied and human-sounding while top-k and top-p prevent genuinely bad tokens from being drawn. Greedy is the special case of sampling where k = 1.