A Vendor Published a Benchmark It Does Not Lead. Here Is What a Buyer Can Read.

On the September 29 snapshot of its own report, Gorgias ranks second in the support lane and third in pre-sale. Gorgias runs that report. It also wrote the rubric, chose the weights and publishes the benchmark in a public repo.

The sponsor is not first here, so the useful question is what the design lets a buyer read. We read the repo, the report and the SaaStr case study that prompted this piece. What follows is our reading of them, including where the design stops.

What the benchmark measures

The repo describes 10-turn themed conversations in two lanes: a Shopping Assistant for pre-sale and a Support AI Agent. Each run is scored on automation rate, answer quality out of 100 (by an LLM judge), latency and a composite of the three.

The report page is titled “Ecommerce AI Agent Benchmark | Gorgias”. When we scraped it on September 29, it had 18 vendors selectable, of which 13 were ranked per lane. So: 18 tracked, 13 ranked. It had measured 208 storefronts of 276 targeted.

Composite weights differ by lane. Shopping uses 40% automation, 35% quality, 25% speed. Support uses 50/40/10, and the repo records that the support weights were reweighted on September 3, 2026. Speed scores 100 at 3 seconds or less and 0 at 22 seconds or more.

Where Gorgias lands

All figures below are the September 29 snapshot, over a 90-day window. The report refreshes weekly, so a later read may differ.

In pre-sale, Envive leads with a composite of 72, Sierra sits second, and Gorgias is third at 63 (plus or minus 2.5).

In support, Decagon shows 77 with an interval of plus or minus 9.9, on 34 conversations. Gorgias shows 68, plus or minus 4.3, on 385 conversations. Yuma shows 67, plus or minus 15.1, on 118 conversations. On support automation alone, Yuma is at 71% against Gorgias at 68%.

Read the intervals before the ranks. Decagon’s lead in support rests on 34 conversations and an interval wide enough to swallow the 9-point gap over Gorgias. Yuma’s interval is wider still. The report prints those numbers next to the composite, so a reader can see how firm each position is. A rank without an interval would hide it.

The disclosure design

Our reading of the repo and report yields six features. This checklist is ours, not Gorgias’s.

• Versioned rubric. Rubric v2.3 lives in the repo. The rubric PDF on the site sits behind a form, so the repo copy is the one a reader can inspect without registering.
• Evidence-forced judge. The judge uses binary checks, and they require quoted evidence, and code verifies each quote against the transcript.
• Blind batches. Vendor and store identity are stripped before judging.
• A second, adversarial pass. It audits a sample of verdicts, not every conversation, and marks a run trusted at 90% agreement or more.
• A ranking floor. A vendor ranks only with a judged quality score and at least 15 conversations in the lane.
• A dated weights change. The September 3 support reweighting is recorded, so a rank movement can be separated from a rules movement.

Two more details matter for reading the numbers. Capture is daily, rankings use a trailing 90-day window, and the trend chart has one point per week. And latency figures are described as “directional”. On our second read, the report says vendors below its coverage bar are not ranked on speed. Our reading is that this can make the ranked vendors look faster than the field. We have not confirmed that the coverage bar equals the 15-conversation floor.

That last item is the most useful in the set. A benchmark that names its coverage rule for speed lets the buyer apply a correction, on our reading.

What the design does not settle

Jason Lemkin’s SaaStr piece is direct about the limit. He writes: “Will there always be some bias? Yes. You wrote the rubric, picked the weights, and chose what to measure. Say that on the first page.” He also writes that SaaStrFund, the fund of the case study’s author, led the seed round for Gorgias. We state both plainly because a reader weighing this benchmark needs them.

Our own reading adds one more. The pages we read state no third-party audit of the rubric, the weights, the judge prompt or the store selection. Gorgias controls all four. The repo shows a last commit on September 27, 2026 and no LICENSE file that we could find, which is why we say public repo and do not say open source.

We make no claim that the judge is unbiased or that the ranking is fair. The evidence supports something narrower: the rubric, the weights, the ranking floor and the dated weights change are written down in the repo, so a buyer can argue with a specific weight.

One caution on secondary reporting. On our second read, the SaaStr piece differs from the report on Gorgias’s support rank and on its pre-sale composite, leaves an unfilled placeholder in the published text, and mixes 18 and 13 vendors. Because the report refreshes weekly, some differences may be snapshot drift. We could not tell which. For that reason we quote figures from the report itself and none from the case study. Lemkin also asserts that buyers’ agents are starting to build vendor shortlists. That is his assertion. For evidence on how agents choose between vendors, see our 42% agreement analysis.

Do this now

Take the vendor benchmark your team leans on most and run the checklist against it. Give each item a yes, no or unknown:

1. Is the rubric versioned where you can read it without a form?
2. Does the judge quote evidence that code checks against the transcript?
3. Is vendor identity hidden from the judge?
4. Is there a ranking floor, and is it stated?
5. Are intervals printed next to every composite?
6. Are weight changes dated?
7. Does it say who chose the rubric, the weights and the sample, and whether anyone outside audited them?

Any unknown is a question for the vendor, in writing. A benchmark that answers all seven can still be self-run. If it does, treat it as a well-documented claim from an interested party and weigh it that way. For related reading, see mining your merge queue for a benchmark and vendor bias in productivity metrics.

---

This analysis synthesizes Everyone Should Publish the Deepest, Most Direct Competitive Evals They Can. Case Study: $100m ARR Gorgias for AI CX (SaaStr (Jason Lemkin), September 2026), gorgias/ai-agent-benchmark (Gorgias, September 2026), and Ecommerce AI Agent Benchmark | Gorgias (Gorgias, September 2026).

Victorino Group helps teams build a vendor-evaluation checklist and apply it before a procurement decision. Let’s talk.