Smarsh Runs the AWS Agentic AI Lens as Code. Missing Evidence Reads 'Not Reviewed'.

The AWS Agentic AI Lens holds 150 best practices, organized into 41 questions across six pillars. Smarsh turned it into a Claude skill that reviews a repository against every one of them, according to an October 2026 post on the AWS Architecture Blog. Each status has to cite evidence found in the repository. An item with no signal is marked Not Reviewed instead of being guessed.

That last rule is the interesting part. In the organizations we work with, governance frameworks live as PDFs, wiki pages or spreadsheets that a reviewer reads and then interprets. The Smarsh version runs. It produces a status for every item, it refuses to invent a status it cannot back up, and the build that assembles its report stops when the output does not match the framework.

A framework that runs, fails closed and is tested by its own eval set behaves like a control. A framework that is only read behaves like a reference.

A caveat before the details. The post is vendor co-marketing published by AWS, and no author is named on the page. By the post’s own account, the lens is not yet selectable as a lens in the AWS Well-Architected Framework. The numbers, in the post’s words, “reflect Smarsh’s first internal uses and the outcomes the team is tracking toward.” Read what follows as a design pattern worth copying, with the evidence still thin.

The lens becomes data

The first move is to treat the framework as a dataset. Each of the 150 best practices has an ID, and the skill reports against those IDs. The status rubric has exactly five values: Compliant, Partially Compliant, Non-Compliant, Not Applicable and Not Reviewed.

A closed vocabulary like this does two jobs. It makes results comparable across repositories and across time, because “mostly fine” falls outside the allowed answers. It also gives the machinery around the model something to check. A free-text review can say anything. A review constrained to known IDs and five statuses can be validated before anyone reads it.

Compare the baseline. On the post’s test fixtures, the model without the skill produced “ad-hoc 18–39 item checklists with no lens traceability.” Those checklists may have contained good observations. Nobody could map them back to the framework, so nobody could tell which of the 150 practices had been looked at and which had been skipped.

The honesty rule

“Every status must cite evidence found in the repository, items with no signal are marked Not Reviewed rather than guessed.”

This sentence carries most of the governance value. A language model asked to fill in 150 rows will fill in 150 rows. Without a rule that separates “I checked and it passes” from “I found nothing,” the report looks complete and quietly overstates coverage. A confident Compliant with no citation is worse than a blank, because the blank at least tells you to look.

Not Reviewed turns the absence of evidence into a visible output. A team reading the report sees exactly where the reviewer had nothing to go on. In our reading, that list is a work queue: some items will need a human to check something outside the repository, others will point at controls that do not exist yet.

The requirement to cite evidence does a second thing. It makes every status auditable after the fact. A reviewer who disagrees with a Compliant can open the cited file and argue with the evidence instead of arguing with the model.

The builder fails closed

The post states it directly: “unknown best-practice IDs or invalid statuses abort the build and hallucinated items cannot silently ship.”

The model can still be wrong, but the pipeline around it blocks results with unknown IDs or invalid statuses from moving downstream. If the model invents a best practice absent from the lens, or returns a status outside the five values, the report does not get built.

This is the same principle we apply to any gate that sits between an agent and production. Validation lives in deterministic code, and the default on failure is to stop. A reviewer that degrades gracefully into plausible output is a liability, because the degraded output looks exactly like the good output.

The skill has its own eval set

Smarsh tested the skill against two synthetic fixture repositories. With the skill, the model passed 16 of 16 expectations. Without it, the baseline passed 8 of 16.

Treat that number with care. Two synthetic fixtures are a small and artificial sample, and both are stand-ins for production repositories. The eval measures whether the skill does what its authors expect on known inputs. It does not measure review accuracy on real code, and the post gives no hours for how long a review takes.

The existence of the eval matters more than the score. A reviewer that has expectations written down can be changed and re-run. When the lens gets a new version, or the prompt changes, or the model changes, the team has a regression check. A governance process run only by people reading a checklist has no such check when the checklist changes.

What it found

Among the gaps the post reports from Smarsh’s own use were a gateway authorizer “running with authentication disabled under a documented risk acceptance” and rate limiting “recorded as a TODO.”

Neither is exotic. Both are the kind of thing a manual review can pass over: a reader sees the risk acceptance and moves on, or sees the TODO and assumes someone owns it. A reviewer that must cite evidence would surface them on each run, pointing at the file. Whether the risk acceptance is still valid remains a human decision. The difference is that the question comes back on each review instead of once.

From report to gate

Smarsh’s stated next step is to “enforce governance by making the review workbook required evidence in Smarsh’s project milestone and gate process before agentic workloads reach production.”

This is the step that converts a useful tool into a control. A report that someone may read is advisory. A report that a milestone requires is enforcement. Note the tense: the post describes this as a plan, and it does not say the gate is live.

The pattern is close to what we described in auditing the review before merge: the gate checks that a review with the right shape exists and holds evidence, and humans keep the judgment calls. The same logic connects to making the agent keep a diary and to giving an AI system an enforcer. A rule nobody executes decays. A rule that runs on every change keeps its shape.

Where humans still sit

The machine produces a status per item, with evidence. In our reading, three responsibilities should stay with people in any version of this design: triaging the Not Reviewed list, deciding whether a documented risk acceptance still holds, and owning the remediation of each Non-Compliant item.

That division is what keeps the reviewer honest in practice. If Not Reviewed items pile up untouched, the report becomes theater. If someone owns that list, it shrinks.

Do this now

Pick one governance framework your organization already claims to follow. An internal security standard, a cloud provider’s well-architected lens, a regulator’s checklist. Then run these four steps.

Convert it to data. Give every item a stable ID and define a closed set of statuses, including one that means “no evidence found.”
Write the honesty rule into the reviewer’s instructions: every status cites a file or artifact, and absence of evidence is reported, never guessed.
Put a deterministic validator between the reviewer and the report. Unknown IDs and invalid statuses stop the build.
Write a handful of fixture repositories with known answers and keep them as a regression set. Run it whenever the framework, the prompt or the model changes.

Only after those four hold should the report become required evidence at a release gate. Making an unvalidated report mandatory gives you a confident document nobody can trust.

This analysis synthesizes Automating agentic AI Well-Architected reviews with the AWS Agentic AI Lens (AWS, October 2026).

Victorino Group helps engineering organizations turn governance frameworks into executable reviews that cite evidence and fail closed. Let’s talk.