Two coding agent tracks each produced a functional BPE tokenizer trainer within 30 minutes. Neither version scaled. The finished artifact, a roughly 2,000-line Rust program called toktoktok that processes trillions of tokens in days on a single 2TB, 128-core machine, took five or more refinement cycles against real constraints to reach. One of the two tracks, running on GPT-5.2, was abandoned before it got there.
That is the whole shape of Mathias Lechner’s writeup of the experiment at Liquid AI, published in August 2026, and it is the most useful account of long-running agent work we have read this year. The toy version is nearly free. The production version is expensive. And the cost lives in a specific place: the loop that carries the agent from the first working draft to an artifact someone else’s system will accept.
Lechner names three ingredients that made the loop converge. Each one is a governance decision wearing an engineering costume.
Ingredient 1: Specify Outcomes and Constraints
The spec that produced toktoktok described what the artifact had to achieve and which constraints it had to respect. It left the mechanism to the agent.
This matters more for long-running work than for single-shot prompts. A mechanism-level spec (“use this algorithm, structure the code this way”) turns the agent into a transcriptionist and puts the human back in the loop for every design decision the spec did not anticipate. An outcome-level spec (“train a BPE tokenizer at this scale, on this hardware, producing this artifact”) gives the refinement cycles something stable to converge toward. The spec survives the rewrite; the mechanism rarely does.
The discipline here is the same one architecture review boards apply to human teams. You review the interface and the invariants. You do not review the for-loops. Teams that write mechanism specs for agents are doing the equivalent of approving every commit line by line, and they will discover the same thing managers discovered decades ago: the reviewer becomes the bottleneck, and the review adds nothing the tests would not catch.
Ingredient 2: A Verification Harness the Agent Cannot Manipulate
The success criterion for toktoktok was external: the artifact must load in tiktoken and in Hugging Face tokenizers. Two third-party systems, neither of which the agent can modify, define what counts as done.
We have documented what happens when agents grade their own homework at scale. The failure is structural: an agent optimizing against a test it can edit will, given enough cycles, edit the test. The countermeasure has been discussed mostly in the abstract. Liquid AI’s experiment is the countermeasure demonstrated end to end. The agent iterated for cycles against a bar it could not lower, and the bar is why the 2,000 lines are trustworthy.
Notice what the external harness buys you that an internal test suite cannot. An internal suite encodes the agent author’s understanding of correctness, and the agent can satisfy the encoding without satisfying the intent. A third-party loader encodes someone else’s understanding, frozen in a system the agent has no write access to. Compatibility with tiktoken is a fact about the world, established outside the loop that is being verified.
For teams running agents on regulated or high-stakes work, this is the transferable pattern, and it composes with the harness engineering practices we described for regulated agentic work. Define done in a system the agent cannot touch. A downstream parser, a production schema validator, a compliance engine you do not own. If every check the agent must pass lives inside the agent’s own writable workspace, you have a loop that converges on passing, which is a different destination from working.
Ingredient 3: Production-Scale Data
Both agent tracks passed on toy data within the first half hour. Per Lechner’s account, the toy datasets masked every critical failure. Nothing about the small-scale runs predicted which designs would survive contact with trillions of tokens.
This is the trap that costs teams the most, because it is invisible at exactly the moment decisions get made. A demo on sample data looks identical to a production system on sample data. The difference only surfaces under real volume, real skew, real memory pressure. In the toktoktok experiment, that surfacing took five or more refinement cycles, working through failures the toy scale had hidden.
The operational lesson is blunt. If your agent pipeline validates against a fixture file, your validation measures the fixture, and the 30-minute toy will sail through it. Budget for iteration against the real workload, on the real hardware envelope, before calling any agent-built artifact production-grade.
Where the Expense Actually Lives
Lechner compresses the economics of the whole experiment into one sentence: “Almost every non-obvious line is cheap to write once you know it needs writing, and the expensive part is knowing that it needs writing at all.”
That sentence should reframe how you budget agent projects. The cost of agent-built software has not gone to zero. It has moved. Writing lines is now cheap enough that both tracks had working toys in 30 minutes. Discovering which lines need to exist is where the refinement cycles went, and that discovery only happened because the harness was external and the data was real. Remove either ingredient and the loop still terminates quickly. It terminates on a toy.
The abandoned GPT-5.2 track is part of the same lesson. A loop with sound ingredients can still fail to converge on a given model. The harness does not guarantee success. It guarantees that success, when claimed, is real, and that failure is visible early enough to stop paying for it.
Do This Now
Take one agent workflow your team currently trusts and answer two questions in writing. First: is the definition of done encoded in a system the agent cannot modify? If every gate lives in the agent’s workspace, move at least one gate outside it this week. A downstream consumer, a third-party parser, a schema the agent cannot edit. Second: what is the largest dataset the workflow has actually been verified against? If the answer is a fixture, schedule one run at production scale and treat every failure it surfaces as a finding, because per Liquid AI’s experience, the toy run was hiding all of them.
This analysis synthesizes Building Production-Grade Agent Loops (Mathias Lechner with Leonie Monigatti, Liquid AI, August 2026).
Victorino Group helps engineering teams design verification harnesses that make agent-built software trustworthy at production scale. Let’s talk.
All articles on The Thinking Wire are written with the assistance of Anthropic's Opus LLM. Each piece goes through multi-agent research to verify facts and surface contradictions, followed by human review and approval before publication. If you find any inaccurate information or wish to contact our editorial team, please reach out at editorial@victorinollc.com . About The Thinking Wire →
If this resonates, let's talk
We help companies implement AI without losing control.
Schedule a Conversation