Agent = Model + Harness: Context Engineering and Evals

There is a clean equation hiding under all the agent hype: Agent = Model + Harness. The model is the weights someone else trained; everything else — tools, scratch files, retrieval, the summarizer, the loop that decides when to stop — is the harness. If you're building an agent and not training the model, you are building the harness. That's the whole job.

Every harness primitive exists to patch a specific deficiency the bare model exhibits: it can reason in text but cannot run arithmetic, remember across turns, or stop itself from drowning in its own history. Bolt on the right primitive and one failure disappears. This guide walks three sides of that problem — harness primitives, the finite attention budget context engineering spends, and the evals that turn it into a flywheel. The companion interactive deep dive lets you toggle each one live.

What is the harness, and why isn't the model enough?#

Start with a bare ReAct loop — think, act, observe, repeat — on a trivial task: read V from file A, double it, write it to file B, report B. The known-correct answer is 2·V; capability is the binary question did the report equal 2·V? With no primitives the trace dead-ends: the model can think "double 7," but there's no executor to run 7 · 2. Add primitives one at a time and each deficiency lifts:

Three primitives, three patches. Filesystem is durable state — anything written survives across turns. Code/bash is a general action, turning a thought into an effect. Sub-agents are context isolation — a side task returns only a condensed result, so the parent's token count doesn't balloon. Each is derived by watching the model fail a specific way, not by asserting a feature. See AI Agent Memory: Beyond RAG to State That Persists for the state-persistence primitive in depth.

Walkthrough: tracing the filesystem + code agent on V = 7#

Watch one concrete run — V = 7, filesystem on, code on, sub-agents off — turn by turn. The known answer is 2 · 7 = 14.

The load-bearing row is Turn 3: transformed = 14, parked in the durable store at Turn 2, is still there when the write step needs it. Rip the filesystem out and rerun the identical input: by Turn 3 the transformed value is gone, and the report comes back wrong. Same model, same task, one missing primitive, opposite outcome.

Without sub-agents, every turn also carries the full history forward, so tokens climb monotonically — the demo flags the context as "contaminated" once low-signal history crowds out the signal, a preview of the next section.

Why does context degrade as it gets longer?#

Context is finite in performance, not just token count. The failure mode — context rot — isn't a mysterious decay constant; it's a direct consequence of how attention normalizes.

Picture needle-in-a-haystack retrieval: one true token sits among N − 1 competitors. Attention scores each position, then runs a softmax into a probability distribution — the attention mass per position:

Retrieval accuracy is the mass that lands on the needle. Because the denominator sums over every position, three levers erode that share — the same three the Chroma "Context Rot" study (July 2025) isolates:

• Length. Add positions and Σ grows, so the needle's slice shrinks — dilution, not decay.
• Distractors. Semantically-close decoys score high (3.0 in the demo) and steal mass directly, hurting far more than plain filler.
• Similarity. A lexical match scores high (up to 6.0) and wins easily; a reasoning-only match scores low and wins on a razor-thin margin.

Layered on top: attention compares every position with every other, so N tokens cost N·(N−1)/2 pairwise relationships — N² growth while the signal thins. The rule: spend the smallest set of high-signal tokens, not the biggest context you can afford.

Pre-loaded RAG vs just-in-time retrieval#

Given that attention budget, how should the harness decide what goes in the window? Two strategies, opposite defaults.

Pre-loaded (classic RAG): rank the corpus and stuff the top-k documents' full bodies up front. Simple and low-latency, but you pay the full token cost of everything loaded, including zero-relevance documents. In the demo's eight-document corpus, the wrong top-k floods the window with dead weight and can even overflow it.

Just-in-time (JIT): load lightweight identifiers up front, then fetch a full body on demand — only for documents whose keywords match. Occupancy and wasted tokens fall far smaller, since you never pay for a body you don't use. The trade is more round-trips against a leaner window; a small corpus can make pre-loading win outright, a large one drowns it. Reranking sharpens either path — see Advanced RAG: Hybrid Search, Reranking & Query Transformation.

Neither is enough alone for long-horizon tasks, so the harness adds compaction: once tokens cross a threshold (say 80% of the window) the harness summarizes the history and continues. Plotting token count produces a saw-tooth — a steady climb, a sharp drop at each compaction — letting an agent run for hundreds of turns without the window rotting.

How do you measure whether the harness is working?#

Evals aren't a report card filed at the end — they're training data for the harness itself, the flywheel. Every failure you measure tells you which primitive to add or tune next. Two design choices dominate.

pass@k vs pass^k. Two metrics with opposite stories, closed-form over k i.i.d. trials with success probability p:

At p = 0.7, the same model tells opposite stories:

At k = 1 they agree; past that they diverge. pass@k makes a 70%-reliable model look near-perfect, since extra tries only add chances to win — the honest metric when one good answer is enough. pass^k makes the same model look brittle, since each extra step is another way to fail — the metric for a multi-step agent where any misstep derails the task. Pick the one your product lives or dies by.

Binary beats Likert. A 1–5 scale invites noise: raters who agree a run is "good" still scatter across 4s and 5s, since the boundary is subjective. Collapse the judgment through a pass/fail threshold and inter-rater agreement climbs, since clear-cut cases snap to unanimous. The demo runs five raters over five transcripts and computes agreement live; binary reproducibly wins — a signal the flywheel can train on, unlike a noisy Likert average.

What to remember#

• Agent = Model + Harness. You build the harness; each primitive is the minimal patch for a watched deficiency, never an asserted feature.
• Filesystem = durable state, code/bash = action, sub-agents = context isolation. Filesystem + code is the minimum pair that completes a persist-then-transform task.
• Context rot is softmax dilution, not decay: length, distractors, and low similarity each steal attention mass, and N tokens cost N²/2 attention pairs.
• Pre-loaded RAG is simple but wasteful; JIT retrieval and compaction (the saw-tooth) keep long horizons lean.
• Report pass@k for best-of-k tasks, pass^k for reliability; prefer binary rubrics over Likert for reproducible agreement.

Keep going#

• AI Agent Memory: Beyond RAG to State That Persists — the durable-state primitive in depth.
• LangGraph Explained: Agents as State Machines — the loop behind the ReAct trace.
• What Is RAG? Retrieval-Augmented Generation from the Ground Up — the pre-loaded baseline.
• Advanced RAG: Hybrid Search, Reranking & Query Transformation — sharpening what enters the window.

Then open the interactive deep dive to toggle primitives live, drag the context-rot levers, and watch pass@k and pass^k diverge.

FAQ#

What does "Agent = Model + Harness" actually mean?#

It's a way of drawing a line around the part you can actually change. You rent the model's raw capability from whoever trained it; the harness is the layer you own, and it's where nearly all of your engineering leverage sits when the weights are off-limits. The useful consequence is that the line moves over time: a job the harness has to scaffold today — keeping arithmetic correct, or carrying a value across turns — can migrate into the model as base models improve, at which point you delete that primitive instead of maintaining it. Treating the split as permanent is the trap; treating it as a moving contract tells you which primitives are worth building now and which are temporary crutches you'll retire.

Why does an LLM get worse with a longer context if it "fits" in the window?#

Because fitting in the window and being attended to are different things. Attention runs a softmax over all positions, so adding tokens — even relevant-looking ones — shrinks the attention-mass share that lands on the token you actually need. Semantically-close distractors steal mass most aggressively, and a needle that requires reasoning starts with a small score and loses on a thin margin. The fix is fewer, higher-signal tokens, not a bigger window.

When should I report pass@k versus pass^k?#

Report pass@k = 1 − (1 − p)^k when a single success is enough and you can take the best of several attempts — it rises with more attempts. Report pass^k = p^k when you need unbroken reliability across every step, like a multi-step agent where any failure derails the task; it falls with more steps. At p = 0.7, k = 4 these read 99.2% versus 24% for the same model — the metric is a choice about what your product actually requires.

Why are binary eval rubrics better than 1–5 Likert scales?#

Because an eval is only useful to the flywheel if two passes of it return the same verdict, and a graded scale quietly leaks that reproducibility. The deeper issue isn't that raters disagree about quality — it's that a five-point scale forces them to also agree on where the cutoffs fall, and nobody shares a definition of what separates a 4 from a 5. Binary drops that second, harder question entirely: a rater only has to decide whether the run cleared the bar, an alignment they can hit far more often. You do pay for it — a pass/fail verdict can't distinguish a barely-passing run from an excellent one — so the right move is to make the bar itself concrete and push any nuance you need into separate named criteria rather than into one fuzzy number.

What is context compaction, and what does it cost?#

Compaction is the harness trading fidelity for headroom: when the running history grows too large to keep affordable, it swaps the raw turn-by-turn transcript for a model-written summary and discards the originals, buying back room to continue. The saw-tooth you see on a token-count chart is simply the byproduct of doing this on a schedule — usage ramps as turns accumulate, then collapses the instant a summary replaces the transcript it stood in for. The shape is the point: an ever-rising line would eventually walk straight into context rot, and each reset is what lets an agent survive hundreds of turns. The catch is that summarization is lossy — anything the summarizer leaves out is gone for good, so overly aggressive compaction can silently drop a detail the agent needed three turns later, which is why what gets summarized matters as much as when.