The Harness Criterion: Keep Every Call In-Distribution

Train a model on short tasks only, wrap it in the right program, and it holds up on held-out tasks 8 to 32 times longer, “with roughly 10x the eval lift with the same train lift over training the underlying Transformer directly.” That result comes from Alex L. Zhang and Omar Khattab of MIT CSAIL, published as research on Zhang’s blog in July 2026. Every experiment runs on one open model, Qwen3-30B-A3B-Instruct-2507.

The number is interesting. The criterion that produced it is more useful, because you can review an existing harness against it in an afternoon.

Locally In-Distribution

The authors name it LID: every individual LM call must see a prompt that is in-distribution with respect to its training data, even when the whole trajectory is out-of-distribution.

Read that as two separate claims about two separate objects. The trajectory is the full arc of an agent run, a four-hour migration across two hundred files, and no training corpus contains anything shaped like it. The call is one forward pass with one prompt. LID says the trajectory is allowed to be strange, and each individual prompt is not.

That decomposition turns a philosophical argument about model limits into a review question with a yes or no answer. Pick any call from a long agent run. Dump the exact prompt that went over the wire. Ask whether that prompt resembles something the model was trained on. If it is a 180,000-token accretion of a system prompt, twelve tool schemas, forty tool outputs, three file dumps, and a plan the model wrote ninety minutes ago, the answer is no, and the model is extrapolating at every step after that.

Zhang and Khattab tie this to scaling directly: “better returns on scaling require compositional generalization, and the capacity for compositional generalization looks like it has to largely live in what today we refer to as a harness.” If you have followed our vocabulary post on what a harness is, this is the strong form of that argument. The harness carries the inductive bias. In their words, “the primary job of the harness should be to carry a higher-level inductive bias that can reduce unfamiliar and complex problems to compositions of simpler ones.”

The Named Diagnosis

The part that will get quoted in engineering channels is the authors’ assessment of the tools people actually run. They write that “existing harness designs such as Claude Code and Codex fail at facilitating locally in-distribution (LID) observations for the underlying neural network… they fundamentally rely on flooding the context window of the Transformer with interleaved task-specific information, tool call outputs…”

The authors reached that by reading the architecture. They did not run Claude Code against a suite and publish a score. They observed that a single growing context, interleaving task text with tool output, guarantees that later calls in a long run see prompts unlike anything in training.

It happens to explain a failure mode most teams have already felt and mislabeled. We have measured how much the harness moves outcomes on identical models; this is a mechanism for part of that swing. Long agent runs degrade in a characteristic way: the first hour is sharp, the third hour produces confident work that quietly contradicts decisions made in the first hour. Teams usually attribute that to context limits or to the model being tired, which is not a real thing. LID gives it a mechanical account. Each successive call is further from the training distribution than the last, and the degradation curve tracks the accumulation, not the clock.

Two Mechanisms Worth Copying

The architecture the authors train is a Recursive Language Model, and two of its mechanisms transfer to harnesses you already run.

The first is context offloading. Input-specific context is passed as a symbolic variable, so the root call never sees the raw content directly. The root model reasons about a handle. Sub-calls resolve the handle when they need the bytes. The prompt the root model sees stays small and stays shaped like the prompts it was trained on, no matter how large the underlying document set gets.

The second is programmatic sub-agent calling with REPL-based storage. Sub-calls are made from code rather than from a text plan, and their outputs land in a store rather than back in the caller’s context window. The parent decides what to read back.

We argued in harness engineering is subtraction that the best harness work deletes scaffolding rather than adding it. Context offloading is that argument with a mechanism attached. The deletion has a destination: a reference, which is how every other layer of computing solved the same problem decades ago.

What the Evidence Covers

Precision about the setup matters here, because the claim is strong and the evidence base is one blog post from two researchers.

Length generalization: 150 training steps at batch size 64 with 4 rollouts per sample using prime-rl, with every 10-step checkpoint evaluated on splits 8 to 32 times longer than the training tasks. Strategy generalization: 500 steps, evaluated every 20 steps on a different environment domain. On MRCRv2, GraphWalks, OOLONG and OOLONG-Pairs, the trained RLM “approaches or exceeds an RLM with a frontier model, GPT-5.5, on the long eval, while far exceeding the base Transformer.”

The authors state the cost themselves: “the runtime of RLM training is 1.5-3x longer than the base Transformer counterparts due to multiple steps per sample and waiting on sub-calls.” That is a real tax and worth quoting to anyone who reads the headline and assumes the approach is free.

Three caveats to carry with the numbers. This is research published on the authors’ own blog, not a peer-reviewed paper, so it has not been through external review. Every result is self-reported. All of it runs on a single 30B open model, so the generalization of the generalization claim is untested.

None of that weakens the criterion. LID is a design principle you can apply and check locally, and it stands or falls on your own harness rather than on their training curves.

Run the LID Review

Four steps, one afternoon, on the agent run in your organization that matters most.

Capture a real prompt. Instrument the harness to log the full prompt for every LM call in a long run. Not a summary. The exact bytes.

Sort by size and look at the tail. Take the largest prompt in the run and read it as though you were the model. Count how many distinct sources of text are interleaved in it. If the answer is more than three, you are outside the shape of ordinary training data.

Find the first offloadable payload. Look for the largest block of task-specific content that the calling model never actually reasons over, and route it through a reference instead. File contents and tool output dumps are usually the first two candidates.

Re-measure the same run. Compare prompt size distribution and end-to-end task success before and after. The failure you are trying to move is the third-hour contradiction, so measure at the third hour.

The strategic consequence for anyone buying AI is worth stating plainly. Reliability on long, unfamiliar work is a property of the architecture you own and can change, so a vendor upgrade does not deliver it and a model swap does not either. The harness is where compositional generalization lives, and the harness is yours.

This analysis draws on Language model harnesses are compositional generalizers (Alex L. Zhang and Omar Khattab, MIT CSAIL, July 2026).

Victorino Group reviews agent harnesses against criteria like LID and rebuilds the ones that flood their own context windows. Let’s talk.