GitHub's Guardrail Lost to One Word. Anthropic Showed Where Control Lives.

An attacker with no credentials and no coding skill opened a public GitHub Issue. The body was plain English: instructions telling the AI agent to fetch a private repository’s README and post it as a comment. GitHub’s Agentic Workflow read the issue, treated its text as a command, and published the private file for anyone to read. Noma Labs documented the run: workflow 23909666039, issue #153, private repo sasinomalabs/testlocal. The exfiltration path carried zero authentication.

GitHub had a guardrail built to stop exactly this. Researcher Sasi Levi got past it by prepending one word. Adding “Additionally” to the injected instruction caused the model to reframe its output rather than refuse. In his words, “by tricking the model, I was able to ensure that GitHub’s guardrails did not work as intended and didn’t prevent the data leak.” A shipped defense, present in production, against a known attack, lost to an adverb.

We wrote about Qwen earlier: post-hoc alignment behaves like a decal, a thin layer you can subtract with a single steering vector, because the underlying capability was never removed. That post ended on an open problem. If suppression is reversible, what would actual removal look like? This week produced a candidate answer, and it arrived in the same seven days as the GitHub failure. The two events map the same conclusion from opposite ends. Control written into the instruction layer cannot hold. It has to move down.

Three layers control can live in

An agent’s behavior is constrained at one of three depths.

The instruction layer is the prompt: system messages, guardrail text, refusal training that tells the model what not to do. This is where most “safety” ships today because it is cheapest to write and fastest to change.

The permission layer is what the agent’s identity is allowed to touch: scoped credentials, least-privilege roles, sandboxed compute. We mapped this territory in the containment stack. Here the model can decide to do the wrong thing and still be blocked, because the environment refuses to execute it.

The weight layer is what the model knows and can do at all. Remove a capability from the weights and no prompt, no jailbreak, no fine-tune reaches it, because it is not there.

GitLost is the instruction layer failing in the field. GRAM, published by AE Studio in collaboration with Anthropic, is the first serious attempt to relocate the off switch into the third.

Why the prompt layer cannot hold

Noma Labs put the mechanism plainly: “the agent’s context window is also its attack surface.” Everything the model reads becomes a candidate instruction, and the model has no reliable way to tell a developer’s policy from an attacker’s note pasted into an issue. The guardrail and the exploit occupy the same channel. A defense that lives in text is a defense an attacker can argue with, and the attacker gets to write in the same language.

We argued a related point in why prompt governance fails at runtime: you cannot prompt an agent into obedience because the prompt is a suggestion the model weighs against everything else in its context. The escalation this week is that GitHub did not forget to add a guardrail. GitHub shipped one, tuned for this attack, and it still lost. The failure is structural, not an oversight.

Noma frames prompt injection as “what SQL injections were to web applications, a systematic, category-wide vulnerability class.” The comparison is exact. We survived SQL injection by moving control out of the string. Parameterized queries stopped concatenating user input into commands. The fix was architectural, at the layer below the text. Agent security is walking the same road.

One clarification on scope. We covered Clinejection, the same attack shape inside a third-party coding tool. GitLost is different in one respect that matters: this is first-party GitHub, the platform itself, not a plugin someone installed. The vulnerability class is not confined to the supply chain. It reaches the vendor’s own agent.

GitHub’s remediation and disclosure timeline are not disclosed in the source. Noma Security sells agentic-AI security tooling and the post ends with a product demo, so read it as vendor research. The proof-of-concept stands on its evidence: the workflow run and issue numbers are reproducible, and the finding does not depend on the vendor’s framing.

What removal from the weights looks like

GRAM, short for Gradient-Routed Auxiliary Modules, attacks the problem the Qwen post left open. During training, it adds a small per-category module to every Transformer layer. When the model trains on dual-use text, virology, cybersecurity, nuclear physics, or in one test a niche programming language, only the matching module is allowed to learn that material. The general-purpose weights stay frozen for that content. The dangerous knowledge routes into a compartment. Delete the compartment after training and the capability is gone.

The efficiency result is the headline for anyone running models in production. Four dual-use categories yield sixteen deployable configurations from one training run. The previous way to get a model without a given capability was to filter the data and train again, sixteen filtered models for sixteen combinations. GRAM produces them from a single run by choosing which modules to keep.

The durability result is the answer to Qwen. Anthropic tested seven model sizes from 50 million to 5 billion parameters. Deleting a module removed the capability about as effectively as never training on the data, with no measured drop in general performance, and the separation between module-on and module-off grew wider as models got larger. Against small-scale malicious fine-tuning, GRAM resisted knowledge recovery about as well as filtering the data out entirely. Post-hoc unlearning did not. Anthropic describes prior unlearning as merely suppressing knowledge, “easy to restore with a small amount of fine-tuning.” That is the exact failure mode the Qwen decal demonstrated. GRAM is the first method that reads like subtraction rather than concealment.

The honesty in Anthropic’s own framing is worth repeating, because omitting it would turn this post into vendor amplification. The results are preliminary. GRAM has never been tested at frontier scale. It has never been applied to any Claude model, and Anthropic writes “we’re not sure it ever will be.” It was evaluated on next-token prediction, not downstream task performance. And some dangerous capabilities may be too entangled with general knowledge for any method to separate cleanly. The off switch is a research result, not a shipped product. What it proves is that the third layer is reachable, not that it is ready.

Anthropic states the goal as three constraints held at once: “limiting access to dual use capabilities in as surgical a way as possible; allowing trusted users to access those same capabilities for beneficial purposes; and do all this without affecting the model’s performance on any other task.” That is a weight-layer specification. None of it can be written in a system prompt.

What to do now

Audit where your agent controls actually live, and reclassify each one by layer.

Take every safety property you rely on and ask which layer enforces it. If a control is a sentence in a system prompt or a refusal the model was trained to produce, mark it instruction-layer and assume an attacker with access to the context window can defeat it. GitHub’s guardrail was instruction-layer. Treat yours the same until proven otherwise.

For every instruction-layer control protecting something real, money movement, private data, code execution, write a permission-layer counterpart that holds when the prompt fails. Scope the agent’s credentials so the private README it was tricked into reading was never readable by that identity in the first place. The containment stack is the blueprint. The interpretability work is how you will eventually verify weight-layer claims, once methods like GRAM leave the lab.

The instruction layer is where control is easy to write and easy to break. The permission layer is where you can hold today. The weight layer is where the industry is heading, and this week it took its first real step. Build for the layer below the one you are on, because the layer you are on is the one attackers are already writing in.

---

This analysis synthesizes An Off Switch for Dual Use Knowledge in AI Models (Anthropic, July 2026), GitLost: How We Tricked GitHub’s AI Agent Into Leaking Private Repos (Noma Labs, July 2026).

Victorino Group helps engineering organizations move agent controls out of the prompt and into permissions and identity that hold when the prompt fails. Let’s talk.