LLM Fine-Tuning, Part 1: When to Fine-Tune & How LoRA Works

Fine-tuning changes the model itself. That is the whole difference between it and prompting or retrieval: a prompt steers a frozen model, RAG feeds it new context at inference, but fine-tuning rewrites the weights with the same gradient-descent machinery that trained the model in the first place — just on far less data, for far less time, at far less cost.

That power is exactly why it is easy to reach for when you shouldn't, and easy to run out of GPU memory when you do. A naive full fine-tune of an 8B model wants 64 GB of VRAM. LoRA drops the trainable part of that bill to a fraction of a percent and lets the same job run on a 16 GB laptop — without changing the math that makes the model better.

This post covers the decision (fine-tune vs prompt vs RAG vs preference tuning), the memory wall that forces LoRA to exist, how LoRA's low-rank adapters actually work, and the one failure mode — catastrophic forgetting — that continued pre-training has to fight.

When should you fine-tune instead of prompting or RAG?#

Post-training is not one tool; it is four, and they solve orthogonal problems. Picking wrong wastes weeks. Diagnose the actual gap first, using four signals scored in [0, 1]:

• Knowledge gap — the model is missing facts about your domain.
• Task-format gap — it knows the facts but won't follow the instruction→response shape you need.
• Preference gap — it does the task correctly but the tone or style is wrong.
• Verifiability — your answers can be auto-checked (math, code, unit tests).

The interactive deep dive turns those four dials and scores each method with a visible formula, so an ambiguous case lights up two methods instead of one:

Read the discounts, because they encode real pipeline order. SFT is discounted by the knowledge gap — teaching a task format on top of missing facts is premature. RLHF/DPO is discounted by the task-format gap — you can't align a model to preferences on a task it can't yet perform. The canonical order falls out of the scoring: CPT/RAG → SFT → RLHF/DPO → RLVR. Add knowledge, teach the task, then align the style, then sharpen verifiable reasoning.

The framework refuses to hand you a single winner when the diagnosis is genuinely split. If knowledge gap and task-format gap are both 0.8, CPT scores 0.80 and SFT scores 0.8 · (1 − 0.25·0.8) = 0.64 — both above threshold, so the honest recommendation is to combine them: continued pre-training to add the domain, then SFT to teach the task. A one-to-one flashcard lookup can't reach that conclusion; a scored ranking can.

Why does full fine-tuning need ~4x the model in VRAM?#

The memory wall is arithmetic, not mystery. To train a parameter you have to hold four things for it, and each has a byte cost:

So full fine-tuning an 8B model in fp16 is roughly 8B × 8 bytes = 64 GB: 16 GB of weights, 16 GB of gradients, 32 GB of optimizer state. That is before activations, which push the real number to 6–8× the model size. A 16 GB laptop is out. An 80 GB A100 fits, barely. At 70B, full fine-tuning wants 560 GB — a multi-GPU job by itself.

QLoRA collapses this. Freeze the base and quantize it to 4-bit NF4 (0.5 bytes/param), so it carries no gradients and no optimizer state. Then train only tiny LoRA adapters. For LLaMA-3 8B at rank 32 targeting the attention projections, the adapters are just 27.26M params — 0.341% of the model — about 55 MB in fp16. The whole training footprint becomes roughly 4 GB (NF4 base) + a few hundred MB (adapters + their grads + AdamW) ≈ 4.2 GB. That fits the laptop. The 70B drops from 560 GB to about 36 GB — a single A100. The optimizer state, full fine-tuning's single biggest line, now covers a rounding error's worth of weights.

How LoRA works: ΔW = B·A#

Full fine-tuning learns a dense update ΔW to a weight matrix — one trainable number per entry. For a DIM × DIM projection that is DIM² parameters. At DIM = 4096 that is 16.78M trainable numbers per matrix, and a transformer has dozens of them per layer.

LoRA's bet (Hu et al., 2021): the useful update lives in a tiny subspace, so you never need a full-rank ΔW. Instead, factor it into two skinny matrices and train those:

The saving scales with model width. At DIM = 4096, r = 32, LoRA trains 2 · 4096 · 32 = 262,144 params instead of 16.78M — 64× fewer, 1.563% trainable. Rank r is the single dial that trades capacity for cost.

Two implementation details make it clean to serve. The applied update is scaled by α/r, which decouples the effective learning rate from the rank so changing r doesn't silently rescale everything:

And B is initialized to zero (A from a small Gaussian), so at step 0 the adapter contributes exactly nothing — the model is the frozen base, then departs gradually as training proceeds. At inference you can fold W + (α/r)·B·A back into a single matrix, so a merged model runs at full speed with zero added latency.

Walkthrough: does a low rank actually recover the update?#

The reason a small r works is spectral: real weight updates are nearly low-rank, so almost all their "energy" sits in a few directions. The best rank-r approximation of any matrix is its truncated SVD (the Eckart–Young theorem), and its error is exactly the tail of the singular values you dropped.

The deep dive plants a known 12×12 update with true rank 3 — singular values σ = [9.04, 5.97, 3.50, ≈0.1, …] — then reconstructs it at each rank and measures the real Frobenius error ‖ΔW − B·A‖:

Watch the knee. Rank 1 can't represent a rank-3 update — the error is huge and the two heatmaps visibly differ. Rank 3 hits the planted rank and the error collapses to near zero; the reconstruction is indistinguishable from the target. Push r past 3 and you gain nothing but parameters, because there is no real signal left, only the tiny noise floor. That spectral gap — a few large singular values and a long flat tail — is precisely what makes LoRA cheap: you buy the top directions and skip the rest.

Continued pre-training and catastrophic forgetting#

Continued pre-training (CPT) is how you add a whole domain: run the same next-token loss on raw domain text. But train on domain data alone and the model overwrites the general English it already knew. This is catastrophic forgetting, and it emerges from the optimizer itself.

You can watch it on a toy classifier. Forward pass p = softmax(W·x + b), cross-entropy loss L = −Σ y·log(p), and the clean softmax+CE gradient ∂L/∂z = p − y; each SGD step moves against that gradient. Pre-train on a general task and it reaches 97% general accuracy, 68% on a new domain. Now continue training:

At 100% domain, every gradient points only at the domain task; the shared weights rotate toward the domain's decision boundary and general accuracy craters. Mix even 20% general data back in and each general example supplies a counter-gradient that pulls the boundary back — held-out general loss stays low because general gradients keep arriving. Data mixing is the cheapest defense. The other two, both visible in the trainer: use a tiny learning rate (roughly 1e-5 for full fine-tuning, 2e-4 for LoRA) so each step overwrites less, and stop early. Crank the learning rate to 2.0 and forgetting accelerates — which is exactly why fine-tuning runs at learning rates orders of magnitude below pre-training.

What to remember#

• Diagnose before you train. Knowledge gap → CPT/RAG. Task-format gap → SFT. Preference/tone → RLHF/DPO. Verifiable reasoning → RLVR. When two signals are strong, combine methods along the pipeline.
• Full fine-tuning ≈ 4× the model in VRAM (weights + grads + AdamW's two moments). That is the wall; 8B → 64 GB, 70B → 560 GB.
• LoRA trains ΔW = B·A, only 2·DIM·r params — 64× fewer at DIM=4096, r=32. QLoRA quantizes the frozen base to NF4 and fits the 8B job in ~4 GB.
• A low rank works because updates are nearly low-rank. Match r to the spectral knee; past it you only chase noise.
• Continued pre-training forgets unless you mix general data back in, keep the learning rate small, and stop early.

Keep going#

• Supervised Fine-Tuning: How a Base Model Becomes an Assistant — the SFT stage that teaches the task format after CPT adds the knowledge.
• RLHF vs DPO: How Models Learn Human Preferences — the preference-optimization stage that fixes tone once the task works.
• RLVR: Reinforcement Learning with Verifiable Rewards, Explained — sharpening reasoning where answers can be auto-checked.
• RL Environments for LLM Agents: Where Models Practice — where verifiable-reward training actually runs.

You can open the interactive deep dive on YeetCode to drag the rank, watch the memory bars fill, reconstruct a real ΔW at each rank, and run the forgetting trainer step by step.

FAQ#

When should I fine-tune instead of using RAG or a better prompt?#

Fine-tune when the gap is behavioral, not informational. If the model is missing facts about your domain, retrieval (RAG) or continued pre-training is cheaper and more truthful — you don't want to memorize a knowledge base into weights. Reach for supervised fine-tuning when the model knows the facts but won't reliably follow your task's input→output format, and for RLHF/DPO when the task works but the tone or style is off. If two of those gaps are large at once, the best system usually stacks methods rather than picking one.

Why does LoRA use two matrices B and A instead of one update matrix?#

Because a single dense update ΔW costs DIM² trainable parameters, while the factored form B·A costs only 2·DIM·r, where r is a small rank. At DIM = 4096, r = 32 that is 262K params instead of 16.78M — 64× fewer. The factorization works because real weight updates are nearly low-rank: their energy concentrates in a handful of singular directions, so a rank-r product recovers almost all of the update while training a tiny fraction of the numbers.

What is the α/r scaling factor in LoRA for?#

The applied update is (α/r)·B·A, and the α/r factor decouples the update's effective magnitude from the rank. Without it, raising r would silently amplify the update because you'd be summing more rank-1 terms. With it, you can change r to trade capacity for cost without also changing the effective learning rate, so hyperparameters stay stable across ranks. Paired with initializing B to zero, it also means the adapter contributes nothing at step 0 — training starts from exactly the frozen base.

How much GPU memory does QLoRA actually save?#

Full fine-tuning holds four buffers per parameter — weights, gradients, and AdamW's two optimizer moments — totaling about 8 bytes, roughly 4× the model size, so an 8B model needs about 64 GB. QLoRA freezes the base and quantizes it to 4-bit NF4 (0.5 bytes/param, no gradients, no optimizer state) and trains only small adapters. For LLaMA-3 8B at rank 32 the adapters are about 27M params (0.34% of the model), so the whole training footprint drops to roughly 4 GB — it fits a 16 GB laptop. A 70B model falls from 560 GB to about 36 GB.

What is catastrophic forgetting and how do I prevent it?#

Catastrophic forgetting is when continued training on new data overwrites capabilities the model already had — train an assistant only on medical text and it can lose general fluency. It emerges directly from gradient descent: if every gradient points at the new domain, the shared weights rotate toward it and away from everything else. The cheapest fix is data mixing — replay general data alongside domain data so general gradients keep arriving and pull the model back. Small learning rates and early stopping help too, which is why fine-tuning runs at learning rates far below pre-training.