Datadog Governs Its Agent Tools With One Rule: No Evals, No Merge

Datadog’s Cloud SIEM team has built more than a dozen MCP tools for its security agent, and the control that keeps the set coherent fits in two sentences from their own engineering post: “Any regression in the baseline eval score greater than 5% is flagged. The runner also flags any tool that ships without eval coverage, which became one of our first governance mechanisms.”

That is the whole enforcement layer. No architecture review board with veto power. No RFC queue. A CI job that refuses to let an untested tool through, and refuses to let a tested one degrade the tools already in production.

Worth saying up front: this is vendor content. The post doubles as a pitch for Datadog Agent Observability and their internal CI eval runner, and every number in it comes from Datadog’s own product telemetry rather than independent measurement. It was published on 17 July 2026 by Chelsea Xu, Eddie Cai, Romain Kirszbaum, and Mohamed Hachem Ouertani, all at Datadog. Read the tooling recommendations with that in mind. Read the governance structure anyway, because the structure is reusable without buying anything.

The Context Window Is a Shared Resource

When several teams contribute tools into the same agent, they are all writing into one finite space: the model’s context window at decision time. Every schema, every description, every parameter list competes for the same attention budget. Datadog states the failure mode plainly: “Add enough tools and the agent can lose track of which one fits the task in front of it.”

That makes the context window behave like a commons. Each team’s contribution is locally rational. The detection team adds a rule-authoring tool because rule authoring is 44% of what its customers do in the product. The triage team adds a bulk-signal tool because roughly 25% of customers triage in bulk and 14% push against the 50-signal UI ceiling. Neither team is wrong. Both additions crowd the same opening, and the cost lands on tool-selection accuracy across the entire set, which no single contributing team measures or owns.

Ordinary code review does not catch this. A new tool can be individually well designed, well documented, and correct, and still make the agent worse at picking among the eleven tools that were already there. The regression is a property of the collection, not of the change.

Five Behaviors, One Weighted Score

Datadog’s answer is to measure the collection on every pull request. The eval framework scores five behaviors:

Did the agent call the tools it should have called?
Did it avoid the ones it should not have called?
Did it use a sensible number of tools?
Did it pass correct arguments?
Did it follow a reasonable trajectory to the answer?

Those roll up into a single weighted score, with tool-selection accuracy weighted highest. Each scenario runs several times and the results are averaged, which damps the non-determinism that makes agent testing awkward in the first place. As they put it: “Traditional tests assume the same input produces the same output, and an agent doesn’t offer that guarantee.” Tool responses are mocked, which keeps output token cost low enough that the suite can run on every change instead of nightly.

The target baseline success rate is roughly 80%, and the reasoning behind that number is the most transferable detail in the post. Eighty percent “leaves room for the occasional hallucination without masking real regressions.” Set the bar at 95% and the suite goes red constantly for reasons nobody can act on, so teams start ignoring it. Set it at 60% and a genuine degradation disappears into the noise floor. The threshold is calibrated to the model’s actual variance, and picking it deliberately is what turns an eval suite into a gate people trust.

Each eval run also generates an experiment in Datadog Agent Observability that localizes which step failed. That part is product-specific. The localization requirement is not: a gate that only tells you the score dropped produces a bisect exercise, and a gate that tells you which scenario and which step dropped produces a fix.

telemetry.intent as a Governance Primitive

Buried in the tool schema is a field called telemetry.intent. The agent must declare, as an input to the call, what it is trying to accomplish.

This is a small design decision with an unusual double life. As observability, it means every tool invocation arrives with the agent’s own stated reason attached, so a trace is legible without reconstructing intent from arguments after the fact. As governance, it converts intent from something you infer into something you can query, alert on, and audit. You can ask which intents the agent declares most often, whether declared intent matches the tool actually chosen, and whether a class of intent should have been refused rather than served.

Compare that with how most agent audit trails work. They record what happened: this tool, these arguments, this response, this latency. Reconstructing why requires reading the surrounding conversation and guessing. We have argued before that an agent’s own record of its reasoning is the audit surface that matters, and a required intent field is the cheapest possible version of that idea: one string, mandatory, at the boundary where the agent asks the system to do something.

If you are designing an internal tool schema this quarter, add the field. It costs a few tokens per call and it is nearly impossible to retrofit onto a year of historical traces.

The Fix for a Confused Agent Is Less Input

The second thing Datadog inverts is what you do when tool performance drops. The instinct is to add: more description, more examples, more parameters, more guidance. Their moves went the other direction.

They broke the schema into named sections (top-level fields, query syntax, options, examples) and made it accept filters, so the agent pulls only the slice it needs. “Leave the filters out and the tool returns everything. Pass them and it returns only the matching slice.” Filtering to a single rule type cut token usage roughly 15%. Adding a detection-method filter reached 41% to 47%.

The bulk-triage redesign is the sharper example. The old tool made the agent hold signal IDs in context. The new one accepts a search query and resolves matching IDs internally: “The IDs never enter the agent’s context, so input token cost stays flat no matter how many signals the query matches.” The redesigned tool handles 500 signals per call, and updating 50 signals dropped from over two minutes to a few hundred signals in a little over a minute.

None of that came from a design meeting. It came from reading behavior: 44% of analyzed messages involved authoring or editing detection rules, 14% of customers hit the 50-signal ceiling, and one customer made seven bulk-triage calls in a nine-minute window, each response exactly 52,366 bytes, moving 350-plus signals in a single sitting. That last data point is the kind of thing no requirements document produces.

Why the Committee Owns Standards and CI Owns Enforcement

Datadog does have a committee. It owns infrastructure and standards, and its stated focus is narrow: “The committee’s main concern is whether a new tool degrades the toolset already in production.” Contribution stays fully self-service through every pre-GA stage.

The reasoning for keeping approval light is worth quoting: “Unlike deterministic features, MCP tools improve mainly through rapid iteration and heavy dogfooding against real use cases, so a heavy approval process would slow the exact feedback loop that makes them better.” Human review is the wrong instrument here because the thing being protected is a numeric property of a set, measurable on every commit. Committees are slow and inconsistent at that. CI is fast and identical every time.

Their honesty about completion is also useful: “no checklist marks an MCP tool as done… the most reliable signal is watching how agents and users behave together in practice.” That is consistent with what we have written about eval difficulty as a product signal. A tool you cannot write a scenario for is usually a tool whose job is not yet defined.

Do This Now

Take your internal agent’s tool registry and answer one question: how many of those tools have an eval scenario, and what happens in CI to a pull request that adds a tool without one?

If the answer is “nothing happens,” you have the same commons problem with none of the instrumentation, and you will not notice the degradation until someone reports that the agent picks the wrong tool. Build the smallest version: three or four scenarios per tool, tool-selection accuracy weighted highest, mocked responses, several runs averaged, a baseline set where your model’s variance actually sits, and a CI check that fails on missing coverage before it ever has to fail on a regression. The coverage check is the part that does the governing. Everything else is measurement.

This analysis synthesizes Creating MCP tools for Cloud SIEM (Datadog, July 2026), written by Chelsea Xu, Eddie Cai, Romain Kirszbaum, and Mohamed Hachem Ouertani. The post is vendor content promoting Datadog Agent Observability, and all figures are the company’s own product telemetry.

Victorino Group helps engineering organizations design eval gates and tool-contribution standards for internal agent platforms. Let’s talk.