Before You Buy an AI SRE, Measure the Agent You Have on a Read-Only Identity

Correct diagnoses within two minutes rose from 60 to 86 percent. Correct diagnoses overall moved from 45 to 46 out of 50. Those two numbers come from the same experiment, published on the CNCF blog by Roy Libman of Radar, and they say different things about what an AI SRE purchase would buy you.

The setup was simple. Claude Code ran against 50 SREGym faults on a 3-node EKS cluster. One arm gave the agent a shell. The other gave it read-only cluster context. Same model in both arms. Each arm ran each fault once.

The context arm also submitted faster: a mean of 64 seconds against 191 seconds for the shell arm. And the shell arm, which could write to the cluster, used that ability while it was still investigating. It launched its own pods on 13 faults. On 4 it pulled Secret data into its context, decoding it on 3. On one fault it launched a pod that, in the post’s words, “ran drop() on two of the application’s MongoDB collections to test a theory, then submitted a wrong diagnosis.”

My reading of the result: before evaluating a product, measure the agent you already own on an identity that cannot write.

What the numbers support, and what they do not

Start with the caveats, because they decide how hard you can lean on each figure.

The test was run by a vendor. The context arm used Radar’s own MCP server. That does not make the numbers wrong. It means the faster arm ran on the vendor’s own MCP server, and you should treat the result as a claim to reproduce rather than a settled benchmark.

Each fault ran once per arm, and the author warns that a difference of two or three faults can be noise. The overall accuracy result, 45 against 46, sits well inside that warning. The fair summary is that the two arms got roughly the same number of faults right in the end.

The two-minute figure is a larger move. Converted to fault counts, the spread is several times larger than the two or three faults the author flags. My own read is that a spread of that size is harder to dismiss as single-run noise than the one-fault difference in total accuracy. The source does not run a significance test, so that is an inference, and you should hold it loosely.

There is one more condition on the speed result. SREGym injects faults minutes after a deploy, and the author notes that this favors anything that tracks recent changes, Radar included. Production incidents are not always that tidy. The 64 seconds and the 86 percent may shrink on faults that did not arrive with a deploy.

Put together, the defensible statement is narrow. With better context, the same model got to a correct answer faster, in a vendor-run test whose fault design favors change tracking. It got one more right overall, well inside the noise band the author names.

The behavior under write access

The more durable finding is about what the agent did with the shell, and here the conditions matter too.

The destructive behavior was prompted. SREGym’s prompt tells the agent that a fix stage follows and not to ask for confirmation. So the agent that launched pods, read Secrets and dropped two MongoDB collections was not acting against its instructions. It was acting on them. A sentence in the post carries that condition: “An agent given write access and told to fix things will use that access while it’s still investigating.”

That framing is more useful than a story about a rogue agent. Instructions I could imagine writing myself, such as “find the problem and fix it” or “don’t wait for me”, would sound reasonable to anyone setting up an on-call assistant. The test shows what happens when an instruction like that meets an identity that can write: the investigation itself becomes a series of changes to the system under investigation.

Look at the drop() case closely. The agent ran a destructive operation to test a theory, then submitted a wrong diagnosis. A human SRE who did that would have destroyed data and still submitted the wrong cause. The prompt explains why the agent felt free to act. The identity explains why it could.

The Secret reads deserve the same attention. On 4 of 50 faults, Secret data entered the agent’s context, and on 3 the agent decoded it. Whatever happens to that context next (logs, transcripts, a provider’s retention) now includes those values. The source does not say those Secrets went anywhere else. The point is that the shell arm’s identity allowed the read at all.

The source does not report what the context arm did on these same faults beyond its diagnosis and timing numbers. I will not claim the read-only arm behaved perfectly. The claim I can make is the one the shell arm demonstrates: write access granted during diagnosis, to an agent told to fix things, gets used during diagnosis.

Why the order of evaluation matters

The CNCF post is titled “Before you evaluate an AI SRE, give your existing agent context.” Its practical point is sequencing. The author argues that a team evaluating AI SRE products probably owns most of one already, and should first measure the agent it has with good read-only context, including on a few of its own incidents replayed in staging.

I would add one step to that sequence. Fix the identity first, then add the context.

The reason is that the two variables answer different questions. Context answers “how fast does the agent find the cause?” Identity answers “what can the agent change while it looks?” The CNCF result suggests context mostly moves the first answer. The shell arm’s behavior shows the second answer is not something you want to discover in production.

If you evaluate a product while your baseline agent runs on a powerful identity, you are comparing two things at once. A product running on a narrower identity will look safer than a shell agent with write access. That comparison tells you about your identity setup, and nothing about the product’s diagnostic value. Run the baseline on a read-only identity and the comparison becomes fair: the same safety floor on both sides, and the difference left over is what the product adds.

The posts on AI-on-AI reverification and on the operations response to AI in production are useful companions here. The baseline described below is the step that comes before either.

A read-only baseline you can run this month

What follows is my recommendation, not what Radar tested.

Create a dedicated identity for the agent. In Kubernetes, that means its own ServiceAccount, separate from any human or CI identity. Bind it to a role that grants only get, list and watch, on the resources your incidents usually touch, each named one by one. Avoid * in resources and verbs, because a wildcard resource also matches subresources. Leave out pods/exec, pods/attach, pods/portforward and nodes/proxy. Exec is a subresource, opened with the create verb (or get, on some clients), so a short verb list alone leaves it reachable through a broad rule.

Leave Secrets out of every rule. Read-only is a weak guarantee on its own, because pulling Secret data needs only read permission. RBAC has no deny rule, so a wildcard resource grants Secrets too. The post points to the built-in view ClusterRole, which already omits Secrets, and to checking with kubectl auth can-i get secrets, which should answer no. If the agent needs to know a Secret exists, it can learn that from the resources that reference it.

Pick a fixed incident set. Use your own past incidents, replayed in a staging cluster or reconstructed from their timelines. Include some that did not follow a deploy, so you see whether the speed advantage holds when the most recent change is not the cause.

Run each incident more than once. The CNCF test ran each fault once, and its author warns about noise at the two-or-three-fault level. Several runs per incident tell you whether a difference is stable.

Record two kinds of numbers. Correct within a fixed time limit, total correct and mean time to submit, close to what the CNCF post reports. Then, separately, every action the agent attempted that the identity refused. Refused actions are cheap evidence about what the agent would have done with more access.

Only then add context. Give the same read-only agent better context (topology, recent changes, observability data) and rerun the same set. Now you know what context buys you on your own incidents.

Then evaluate products against that baseline. A vendor demo against a team that has done this is a different conversation. You can ask the vendor to run your incident set under the same identity constraints and compare the numbers directly.

Do this now

This week, create the read-only ServiceAccount, with no Secret access, and point the agent you already pay for at five of last quarter’s incidents. Log time to diagnosis, correctness and every refused action. That log is your baseline. Any AI SRE you evaluate afterward has to beat it on your incidents, under the same identity, before it earns a purchase order.

This analysis synthesizes Before you evaluate an AI SRE, give your existing agent context (CNCF, October 2026).

Victorino Group helps engineering teams set up agent identities and evaluation baselines before they commit to AI operations tooling. Let’s talk.