Five Questions to Ask a Harness Vendor, Scored Against the Biggest One Shipped This Month

On 1 September, Kai Waehner published a five-question test a buyer can run in a vendor meeting to find out how portable an agent harness actually is. On 10 September, OpenAI shipped the Agents API, a managed harness runtime that OpenAI describes as “fully managed by OpenAI”. Nine days separate the test from the largest available subject for it.

We have already argued that the harness is where governance now lives and that it is a short-lived artifact both ends of the stack are eating. This piece does something narrower. It takes a named checklist, runs it against a shipped product, and reports which questions the announcement answers.

The five questions

Waehner’s test, stated as questions you can read off a page:

Can you export agent definitions in a format another system can read, and has anyone actually done it?
Where does accumulated context live, and can you point at it in your own data layer?
Could the whole thing be re-hosted on infrastructure you control?
Do you get the full audit trail, or only the vendor’s summary of it?
What is the realistic exit cost in engineer-months, measured today?

His procedural instruction matters as much as the content: “Ask all five of the same vendor, in the same meeting, and write the answers down.” One question at a time, spread across a quarter, gets five reassuring answers that never have to agree with each other.

Above the checklist he puts a single tiebreaker: “does the context your agents accumulate live in your data layer or inside the harness?” His piece is an argument. It reports no benchmark and no survey, and it should be read as reasoning.

Waehner also separates three shapes of harness by how much the vendor owns. A framework harness lives in your code, and you maintain all of it. A managed runtime harness puts execution, state and scale with the vendor, where “your agent definitions live in their format”. An embedded harness sits inside an application platform, and “the agents work only where they were born”. He declines to rank them: “None of these is wrong. They are three different deals, and the mistake is signing one without reading it.”

Scoring the Agents API announcement

What follows scores one document, the launch announcement, not the product. A raw-text read of that announcement found no approval gates, no human-in-the-loop, no audit trail, no permissions or RBAC and no policy engine. The only control-shaped surfaces in it are sandbox isolation, vault_ids for secret scoping, max_concurrent_subagents as a blast-radius ceiling, and VPC or self-hosted deployment. Silence in a launch post is silence. The product may hold more than the text says, and this text is what a buyer has to work with before the meeting.

Question 1, export. The announcement is silent on exporting agent definitions. The underlying harness is the open-source Codex harness, published at github.com/openai/codex, and the announcement says that “The Agents API provides versioned access to these capabilities with each model launch.” Whether a definition running on the managed runtime comes back out in a form another runtime reads is not addressed.

Question 2, context. Here the announcement speaks, and it speaks for the harness. OpenAI’s framing: “Useful agents need a powerful harness that manages context, uses tools efficiently, and coordinates subagents.” The shipped features are automatic compaction as a session nears its context limit, tool search that loads tool definitions on demand, programmatic tool calling, and subagents each carrying their own context. Long Lake CTO Rasmus Wissmann, quoted in the same announcement, says the opposite way round: “Agents API supplies the harness; the environment, context, and UX stay ours.” Both sentences are in the launch post. Waehner’s tiebreaker asks a buyer to resolve exactly that, with a pointer to a table in their own data layer.

Question 3, re-hosting. This is the question the announcement answers best. Three environment choices are offered: an OpenAI-managed sandbox, a partner sandbox, or your own infrastructure or VPC. The named partners are Blaxel, Cloudflare, Daytona, DigitalOcean, E2B, Modal, Oracle, Runloop and Vercel. Execution placement is genuinely negotiable. Question 3 is about where the workload runs, and question 1 is about whether the definitions travel; a good answer to one does not supply the other.

Question 4, audit trail. Silent. Waehner’s own framing explains why that is expected: the harness is the inner loop, one agent’s reasoning cycle with its tools, memory and guardrails. Workflow orchestration is the outer loop, “coordinating work across agents, systems, and humans, with approvals, schedules, retries, lineage, and the audit trail that regulated industries require.” A buyer who expects the harness vendor to hand over the audit trail is asking the inner loop for an outer-loop artifact.

Question 5, exit cost. Silent, and structurally so. The pricing statement covers entry: “There are no additional fees for using the Agents API, you simply pay for the tokens and tools your agents use.” Nothing in the announcement estimates the cost of leaving. That number is the one a buyer has to produce themselves, in the meeting, in engineer-months.

One more caution on the announcement’s numbers. Every performance figure in it is a customer testimonial with no stated method: an evaluation score moving from 0.71 to 0.85 and a 4x latency reduction (Ciridae), a 60% reduction in cost per case (SafetyKit), an 86% reduction in failed agent responses (Hypha). OpenAI published no benchmark of its own alongside them.

Why question 5 is the expensive one

A paper submitted to arXiv on 8 September gives the first measured result we have seen on what happens when the harness and the model are adapted against each other. The authors define a harness as “the system prompt, tool set, execution hooks, and context-management scaffolding around a model”, then fine-tune a weaker model on a stronger model’s trajectories under a harness that had already been evolved for that weaker model. From the abstract: “performance regresses on all seven tasks by 4 to 30 points across Qwen3-Coder and Gemma 4, even though the same procedure helps under the unevolved harness.”

The stated cause is fit. The knowledge did transfer. Imitation “transfers knowledge and increases scaffold usage, but disrupts model-harness fit: the weaker model adopts the expert’s planning strategy without the competence to execute it and no longer matches the harness evolved around its native planning style.”

Their fix keeps the model’s own style: “an on-policy expert-correction pipeline, automated by a meta-level MLE agent, that localizes the failing turn in the weaker model’s own rollout and asks the expert to rewrite only that turn.”

A buyer does not need to run that pipeline to use the finding. It says the harness and the weights are coupled objects, and that changing one under an arrangement tuned for the other can cost more than it returns. That is what converts “we could migrate later” into a number.

It also sharpens Waehner’s investment guidance, which cuts against the instinct to build the harness up: “Keep the clever logic thin and disposable. Invest in the durable parts instead: state, policy, audit, and the portability of your definitions.” His reason is that “every model generation changes the optimal agent structure, and over-engineered control flow breaks with the next release.”

A vendor who answers question 4

Spacelift Flows, announced 8 September, is worth reading next to the Agents API because its post lists the controls the other one does not mention. Spacelift claims approval gates, multi-stage sign-offs across dev, staging and production, a step-by-step audit trail from the first run, Slack approvals routed automatically off infrastructure events, and an MCP server that gives agents “their own governed path”, so flows expose a curated set of approved tools instead of raw infrastructure access.

Two caveats. The post is written by a technical product marketing manager and publishes no numbers at all, so treat the list as claimed product surface rather than verified behaviour. And Spacelift is not competing with a harness. It sits in Waehner’s outer loop, which is where he places approvals and lineage in the first place. The useful reading is that the two documents describe different floors, and a buyer needs both answered by someone.

We wrote earlier about a lab turning the harness into a SKU and about what a provider-owned session leaves out of your records. The Agents API announcement is consistent with both.

Do this now

Book one meeting with your current or prospective harness vendor. Ask all five questions in that meeting, write the answers down, and treat a missing answer as an answer.

Then attach one artifact to each:

Export. Ask for one exported agent definition, today, as a file. Try loading it somewhere else.
Context. Name the table, bucket or index in your own systems where accumulated agent context lands. If you cannot name it, the harness owns it.
Re-hosting. Get the deployment options in writing, including the partner or VPC path, and confirm which of them your security review already approves.
Audit trail. Decide who owns the outer loop before you buy the inner one. If nothing in your stack currently logs approvals and lineage across agents, that is the purchase to make first.
Exit cost. Write a single number in engineer-months, with the assumptions beside it, and date it. Re-measure it in a quarter and compare.

Waehner’s closing rule is the one to carry into the room: “Choose the model for capability. Choose the harness for what it lets you keep.”

This analysis synthesizes The AI Agent Harness: Where Vendor Lock-in Went After the Model Became Swappable (Kai Waehner, September 2026), Introducing the Agents API (OpenAI, September 2026), Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails (Zhou Yu, Bin Bi, Shiva Kumar Pentyala and co-authors, September 2026), and Spacelift Flows is live (Spacelift, September 2026).

Victorino Group helps engineering organizations run harness procurement with portability and audit ownership decided before the contract. Let’s talk.