Supervised Fine-Tuning: How a Base Model Becomes an Assistant

A base language model is an autocompleter. Feed it What is EBITDA? and it will happily continue with another question, because the pretraining data is full of FAQ pages where questions follow questions. It has the knowledge. It does not have the manners.

Supervised fine-tuning (SFT) is how you install the manners. You show the model a few thousand (instruction, response) pairs and run the ordinary training loop over them — forward pass, loss, backward pass, weight update. Nothing exotic. The loop is byte-for-byte the same one used in pretraining. Two small changes turn it into instruction-following: the data gets wrapped in a template, and the loss gets masked so the model is scored only on the response, never on the prompt.

The whole thing fits in one training step you can compute by hand. This walks through that step on a real 25-token toy model — the same one behind YeetCode's interactive deep dive — so every number below (3.205, p ≈ 4%, the loss dropping to 2.71) is arithmetic, not a metaphor.

The one thing SFT changes#

Continued pretraining (CPT) and SFT share an identical forward and backward pass. The only difference is which tokens carry a loss.

CPT scores every token. It teaches the model to reproduce the entire sequence — the question and the answer — so it keeps learning language, not instruction-following. SFT instead sets the label of every prompt token to the sentinel value -100. PyTorch's cross_entropy(logits, labels, ignore_index=-100) skips those positions: they contribute exactly 0 to the loss and are dropped from the average. The gradient only ever pushes on the response.

That single line — labels[:response_start] = -100 — is the entire mechanical difference between "learn to talk like this corpus" and "learn to answer when asked."

One real training step, end to end#

Take one pair: instruction What is EBITDA?, response EBITDA stands for Earnings. Wrapped in the Alpaca template and tokenized, it becomes a 16-token sequence, which gives 15 next-token targets (every position after the first predicts the next one).

The step runs in five phases:

1. Tokenize. The (instruction, response) text is formatted into one template string and split into token ids.
2. Mask. The 10 prompt targets get label -100. The 5 response targets stay scored.
3. Forward + softmax. For each scored position, the model produces logits over the 25-token vocabulary; softmax turns them into a probability distribution that sums to 1.
4. Cross-entropy. Each scored position costs −ln(p_target) — the negative log of the probability the model gave the correct next token. A fresh model spreads its mass roughly evenly, so p_target ≈ 1/25 ≈ 4% and each loss is −ln(0.04) ≈ 3.2.
5. Gradient step. Backprop, then step against the gradient: W -= lr * grad. Re-evaluate: the loss drops.

The softmax cross-entropy gradient is famously clean: dL/dlogits = softmax(logits) − onehot(target). Subtracting the one-hot pulls probability toward the correct token and away from everything else. Stack thousands of these steps over thousands of pairs and the autocompleter becomes an assistant.

Walkthrough: masking the EBITDA sequence#

Here is the full 16-token sequence with its 15 targets. A target at position t is masked when t - 1 still sits inside the prompt span (which ends at index 10, the : after Response). Everything up to and including that : is context the model reads but is never graded on.

The masked positions still run a forward pass — the model has to condition on them to predict what comes next — but their prediction error is thrown away. Only rows 11–15 enter the arithmetic.

The batch loss is the mean over scored targets only: sum of 5 losses ÷ 5 = 3.205. Not ÷ 15. Masking changes both the numerator and the denominator, which is why the reported loss is a pure property of the response. Forget the ignore_index and you silently divide by 15 instead of 5 — a common bug that makes SFT and CPT look identical when they are not (CPT on the same pair reports 3.290 over all 15 targets).

Now the gradient step. With learning rate 1.0, one update raises the probability the model assigns to each response token, and re-evaluating gives:

Push the learning rate to 40 and the step overshoots the minimum instead of descending into it — the loss rises, 3.2 → 5.3 → 27.6. That is real divergence, not a glitch, and it is why SFT runs use small learning rates with warmup.

Learning to stop: EOS as an argmax decision#

The single capability a user feels most is that the assistant stops talking. That is not a special "done" flag — it is the model assigning enough probability to the <eos> token that greedy decoding (argmax at each step) actually picks it.

Watch P(<eos>) right after the model has generated Earnings, as more gradient steps land on the pair:

The base model puts almost no mass on <eos>, so argmax never selects it and generation runs to the cap. A handful of real steps climb P(<eos>) past its competitors, argmax finally picks it, and decoding halts. "Knowing when to stop" is just probability mass moving onto one token.

Alpaca and ChatML: the templates that carry the mask#

The template is what tells the loss mask where the response begins. Two formats dominate.

Alpaca is plain text with header sections:

Everything up to and including ### Response: is the prompt span (masked). The tokens after it, plus the <|endoftext|> end marker, are scored.

ChatML uses explicit role tokens instead, which is what most modern chat models are trained on:

Here the assistant turn is closed by <|im_end|>, which doubles as the chat EOS. The mask boundary sits right after <|im_start|>assistant — everything before it is context, everything after is graded. Same masking rule, different delimiters. Pick one and stay consistent, because the model will learn the delimiters as literally as it learns the answers.

When SFT goes wrong#

Failures are not anecdotes — they fall out of what the training distribution did or did not put gradient on.

Format overfitting. Train on exactly one template and the model bonds to the delimiters, not the task. Probe it with a slightly different prefix and the answer's probability mass collapses. The fix is template diversity, not more epochs.

Hallucination. Ask about something never covered by the response tokens and the model has no learned "I don't know" behavior — SFT only reinforces the answers you showed it. The output distribution stays flat and confident, with no refusal, because you never put gradient on refusing.

The alignment tax. Fine-tune narrowly and general ability quietly decays. Training the toy model on finance-only data and measuring a held-out general-domain loss shows the tax directly — and shows the standard mitigation, mixing a slice of general data back in:

At 0% mix, the finance loss drops beautifully while the held-out general loss more than doubles — catastrophic interference. Blending even 20% general data back in nearly erases the tax at a small cost to finance sharpness. That is the empirical basis for the common "80/20" rule: keep roughly a fifth of the fine-tuning mix general-purpose.

Why does so little data work at all? The Superficial Alignment Hypothesis: the base model already knows the facts from pretraining. SFT is not teaching new knowledge — it is teaching an interface, selecting which of the model's existing behaviors to surface. That is why a few thousand high-quality pairs move the needle far more than a million mediocre ones.

What to remember#

• SFT is the pretraining loop with two edits: a chat template and a loss mask (labels[:response_start] = -100).
• Only response tokens are scored; masked prompt tokens contribute 0 and leave the mean's denominator, so the loss is a pure measure of the answer.
• Per-token cost is −ln(p_target); the gradient is softmax − onehot; too large a learning rate makes the loss rise instead of fall.
• Stopping is learned — <eos> just needs enough probability mass for argmax to pick it.
• Narrow fine-tuning taxes general ability; mix in ~20% general data to pay it down.

Keep going#

• RLHF vs DPO: How Models Learn Human Preferences — the next stage, where the model learns from preferences rather than single gold answers.
• RLVR: Reinforcement Learning with Verifiable Rewards, Explained — training on rewards you can check by machine.
• RL Environments for LLM Agents: Where Models Practice — where post-SFT models rehearse multi-step behavior.
• From APIs to Agents: Function Calling and the ReAct Loop — turning an instruction-follower into a tool user.

Or open the interactive deep dive and drive the real training step yourself — edit the pair, mask or unmask the prompt, drag the learning rate until the loss diverges, and watch every number recompute live.

FAQ#

What is the difference between supervised fine-tuning and pretraining?#

They run the identical loop — forward pass, cross-entropy loss, backprop, weight update. Pretraining (and continued pretraining) scores every token, teaching the model to reproduce whole sequences of raw text. SFT wraps the data in an instruction template and masks the prompt tokens with the label -100, so the loss and gradient come only from the response. Pretraining teaches language; SFT teaches the model to answer when asked.

Why are prompt tokens labeled -100 in SFT?#

-100 is PyTorch's default ignore_index for cross-entropy. Any target with that label is skipped entirely: it contributes 0 to the loss and is excluded from the mean's denominator. This keeps the model from being trained to generate the user's question — you only want gradient on the assistant's reply. Concretely, for a 15-target sequence with a 5-token response, masking scores 5 targets and averages over 5, not 15.

How much data does supervised fine-tuning need?#

Far less than pretraining — often a few thousand high-quality pairs. The Superficial Alignment Hypothesis explains why: the base model already learned the facts and reasoning during pretraining, so SFT only has to teach the format of being a helpful assistant, not new knowledge. Quality and diversity beat volume; a small curated set outperforms a large noisy one, and over-narrow data triggers the alignment tax.

What is the alignment tax and how do you avoid it?#

The alignment tax is the drop in general ability that happens when you fine-tune narrowly. Train only on finance data and a held-out general-domain loss can more than double (in the toy model, 0.776 → 1.630) even as the finance loss falls — the new gradient overwrites unrelated capabilities. The standard fix is to mix general-purpose data back into the fine-tuning set. Blending in about 20% general data nearly eliminates the regression, which is the origin of the practical 80/20 rule.

Why does a fine-tuned model know when to stop generating?#

Because it learns to assign high probability to the end-of-sequence token (<eos> or <|im_end|>) after a complete answer. Base models put almost no mass there — around 4% — so greedy decoding never selects it and the model rambles to a length cap. After a handful of gradient steps that include <eos> as a scored target, its probability climbs past every alternative (0.042 → 0.729 in a few steps), argmax finally picks it, and generation halts. Stopping is an ordinary next-token prediction, not a special control signal.