RLVR: Reinforcement Learning with Verifiable Rewards, Explained
How RLVR trains LLMs to reason — test-time compute, verifiable rewards vs. hackable reward models, and GRPO's group-relative advantage, from o1 to DeepSeek-R1.
This article has an interactive companion
Don't just read it — step through it interactively, one state change at a time.
For years, improving a language model meant one thing: make it bigger, feed it more data, burn more training FLOPs. Then o1 added a fourth knob — let the model think longer at inference. Accuracy on hard math kept climbing the more tokens the model spent reasoning, with no retraining.
The surprising part wasn't the thinking but how it got there. DeepSeek-R1 showed you can grow this behavior with reinforcement learning driven by a reward you can compute — is the answer right, yes or no — instead of a learned reward model that guesses human preference. That swap, plus an optimizer that throws away half of PPO's moving parts, is RLVR: Reinforcement Learning with Verifiable Rewards.
This post walks the mechanism end to end: test-time compute as a scaling axis, verifiable rewards versus hackable reward models, GRPO's critic-free advantage, and the capacity floor. You can explore each piece interactively on YeetCode alongside the math.
Why is test-time compute a fourth scaling axis?
The three classic axes are parameters, data, and training compute. Test-time compute is different: you spend it after training, per query, by letting the model emit a long chain of reasoning tokens before committing to an answer.
On AIME-2024 (a hard competition-math benchmark), a fast non-reasoning model lands in the low teens; give a reasoning model thousands of thinking tokens and accuracy climbs toward 80%+ — the deep dive fits that curve across GPT-4o, o1-preview, and o1.
But three quantities that look linked are three different functions:
| Quantity | What it tracks | Formula (as the deep dive computes it) |
|---|---|---|
| Accuracy | Thinking budget, saturating | L / (1 + e^(−k·(log10 tokens − x0))) |
| Cost | Total tokens generated | (tokens / 1000) × price_per_1k |
| Latency | The longest single chain | chain_len / decode_rate |
The gap opens under best-of-N. Run N chains in parallel and keep the best: you pay for all N (cost is linear in total tokens), but latency tracks only the longest chain — roughly 1/N of the serial time. So "just think longer" carries a real, tunable price tag, not a free lunch.
System 1, System 2, and the "aha moment"
Pre-o1 LLMs were pure System 1 — fast, intuitive, one forward pass to an answer. Reasoning models learn slow System 2: draft, check, backtrack, retry.
The remarkable part is that self-correction emerged. Nobody wrote "when you suspect an error, stop and reconsider" into the training data; under RL that rewards only correct final answers, the model found on its own that pausing mid-solution — "Wait… let me re-check that. Actually…" — raised its hit rate. It was never demonstrated, only incentivized.
That is the thesis of reward-driven reasoning: don't teach the procedure, reward the outcome and let search over the model's own outputs surface it. Which makes the reward signal the most important design decision in the system.
Write a verifier, not a reward model
Standard RLHF scores a response with a reward model — a second network trained to predict human preference. It works for fuzzy goals like "be helpful," but because it is learned, it is hackable: the policy finds inputs the reward model scores high but a human would score low. That's reward hacking, and it's why RLHF needs a KL leash to keep the policy out of the model's blind spots.
For anything with a checkable answer — math, code, structured output — you don't guess preference; you write a program that decides correctness. DeepSeek-R1's reward is exactly this, two deterministic terms:
reward = 1.0 · is_correct(answer) + 0.1 · has_format(<think>…</think>)Correctness (1.0) drives capability; format (0.1) nudges the reasoning into parseable tags — so a formatted correct answer scores 1.1, a wrong one 0.
But "verifier" is not a synonym for "trivial." The playground lets you game three toy verifiers:
- A math verifier that grabs the last integer scores
1.0for"it's definitely not 391, it's 391"— right token, wrong reasoning. - A code check that pattern-matches the word
truepasses"true true true"with zero logic. - A JSON verifier that only asserts parseability accepts
{"anything": 0}when the task demanded{"ok": true}.
A shallow verifier is a reward model in disguise — the policy optimizes the extractor, not the task. A real one parses a structured answer field, runs hidden tests, or asserts the schema. Get it right and reward hacking mostly disappears: only the true answer scores.
GRPO: group-relative advantage without a critic
RL needs to know whether a response was better than expected. PPO answers with a value model (a critic) — a second trainable network estimating the baseline reward for each state, expensive and noisy to train.
GRPO (Group Relative Policy Optimization) deletes the critic with one idea: the group is the baseline. Sample G responses to the same prompt, score each, and measure every one against the group's average.
function groupAdvantage(rewards) {
const G = rewards.length;
const mean = rewards.reduce((a, b) => a + b, 0) / G; // baseline, no critic
const dev = rewards.map((r) => r - mean); // beat / miss the group
const variance = dev.reduce((a, d) => a + d * d, 0) / G;
const std = Math.sqrt(variance);
const A = rewards.map((r) => (r - mean) / (std + 1e-6)); // zero-mean, unit-std
return A; // sign of A survives the clip; magnitude only scales it
}The advantage is just a z-score over the group:
A_i = (R_i − mean(R)) / (std(R) + ε)Subtracting the mean makes the signal relative — only "better or worse than this prompt's typical attempt" survives. Dividing by the std keeps the step size stable whether the prompt is easy (rewards bunched) or hard (rewards spread). The ε ≈ 1e-6 guard matters — more below.
Walkthrough: normalizing a group of four rewards
One prompt, G = 4 responses, verifier scores:
- Response A — correct and formatted →
1.1 - Response B — format only, wrong answer →
0.1 - Response C — correct but no
<think>tags →1.0 - Response D — wrong, no format →
0
Trace the exact normalization, one stage at a time:
| Stage | Computation | Result |
|---|---|---|
Rewards R | verifier scores the group | [1.1, 0.1, 1.0, 0] |
| Mean (baseline) | (1.1 + 0.1 + 1.0 + 0) / 4 | 0.55 |
Deviations R−mean | subtract the baseline | [+0.55, −0.45, +0.45, −0.55] |
| Variance | mean of squared deviations 1.01 / 4 | 0.2525 |
| Std | √0.2525 | 0.5025 |
Advantage A | (R − 0.55) / (0.5025 + ε) | [+1.09, −0.90, +0.90, −1.09] |
Read the last row as the training signal. Response A (+1.09) and Response C (+0.90) beat the group, so the policy is pushed toward them; B and D are pushed down. A got the strongest pull, sitting furthest above the 0.55 baseline — and a single average, not a critic network, produced all of it.
The edge case teaches the most: make every response score the same. All correct ([1.1, 1.1, 1.1, 1.1]) or all wrong ([0, 0, 0, 0]) gives std = 0, the ε guard drives every advantage to ≈ 0, and the gradient vanishes — a dead batch with nothing to reinforce. Hold that thought.
The clipped objective and the ~1.5B capacity floor
The advantage feeds a PPO-style clipped-surrogate loss, and GRPO keeps the KL leash even after dropping the critic:
L = − mean( min( ρ·A_i , clip(ρ, 1−ε, 1+ε)·A_i ) ) + β·KL(π_new ‖ π_ref)Here ρ = π_new(o_i) / π_old(o_i) is the importance ratio. The min + clip cap how far one update can move on a single sample, so the policy can't lurch: what survives the clip is the sign of A_i — reinforce vs. suppress. The β·KL term anchors the policy to a reference model so it stays fluent instead of collapsing into reward-gaming gibberish.
The memory payoff is concrete. PPO keeps four models resident; GRPO keeps two plus a zero-parameter verifier. Under Adam mixed-precision a trainable model costs ~16 bytes/param, a frozen bf16 model 2. For a 7B base:
| Slot | PPO | GRPO |
|---|---|---|
| Policy (trainable) | 112 GB | 112 GB |
| Critic / value (trainable) | 112 GB | — dropped |
| Reward model (frozen) | 14 GB | — replaced by verifier (0) |
| Reference (frozen) | 14 GB | 14 GB |
| Total | ~252 GB | ~126 GB |
Half the footprint, with the critic and reward model both gone.
But RL is not magic. It only amplifies what the base model can already do. If the base essentially never solves the task, every response scores 0, std = 0, and every batch is dead forever. This isn't a formula but an empirical floor observed around ~1.5B parameters:
| Base model | Size | Outcome |
|---|---|---|
| SmolLM-135M | 0.135B | Real GRPO run → all rewards 0, dead |
| Qwen-1.5B | 1.5B | R1-style reasoning reproduced |
| DeepSeek-R1-Zero base | 671B | Reasoning emerged from pure RL |
RLVR sharpens latent capability; it cannot conjure it from nothing.
Keep going
- RL Environments for LLM Agents: Where Models Practice — where the rollouts and rewards come from.
- From APIs to Agents: Function Calling and the ReAct Loop — how a reasoning model turns thoughts into actions.
- Agent = Model + Harness: Context Engineering and Evals — the scaffolding a trained model runs inside.
- AI Agent Memory: Beyond RAG to State That Persists — giving a reasoner state that outlives one turn.
Then open the interactive deep dive to slide the thinking budget, game a verifier, watch the z-score compute live, and push the base model below the capacity floor until the run goes to all-zeros.
FAQ
What is the difference between RLHF and RLVR?
Different domains, not rivals. RLHF handles goals no program can score — tone, helpfulness, taste — accepting a learned proxy and paying for it with reward hacking and a KL leash. RLVR applies only where correctness is machine-checkable: math, unit-tested code, structured output. Frontier systems usually run both. The quick test for which to reach for: could you write an assert that settles the answer? If yes, verify; if not, you're stuck with preference.
How does GRPO work without a value model?
A baseline in policy-gradient RL exists for one purpose — cutting variance. Subtract anything independent of the action and the gradient stays unbiased but quieter. PPO learns that baseline with a critic network; GRPO reads it straight off the siblings, since a batch sharing one prompt has a valid action-independent baseline in its own mean reward. You trade a trained critic for a few extra rollouts — and reasoning runs already sample widely to find rare correct chains, so those rollouts are nearly free.
Why does the DeepSeek-R1 reward use 1.0 and 0.1?
Their ordering matters more than the exact numbers: format must be worth strictly less than correctness and strictly more than zero. Set format to 0 and early training stalls — most rollouts are wrong, score a flat 0, and the group has no spread to normalize. Push it too high and the model farms points by wrapping wrong answers in perfect tags. 1.0 versus 0.1 is the safe middle: enough shaping to keep gradients flowing, not enough to rival getting the answer right.
What is the ~1.5B parameter capacity floor?
Not a hard constant on the parameter count — it's where a model's success rate on a task falls to effectively zero, leaving RL no spread of outcomes to learn from. The threshold slides: easier tasks or a curriculum of solvable problems push it down, olympiad-hard benchmarks push it up. You can also sidestep it by seeding capability first — a short supervised fine-tune on a few reasoning traces, or distillation from a bigger model — so some rollouts already land. The ~1.5B figure is a reported landmark for R1-style math, not a law of nature.
Make it stick: run this one yourself
Don't just read it — step through it interactively, one state change at a time.