· Court Transcript Platform
Rev on our benchmark, and a number for who said it
We put Rev.com's transcripts of the same two Supreme Court arguments through our own benchmark, added the attribution metrics legal work actually turns on, and measured whether feeding the engines the case caption helps. Two of three results flatter us. The third is the roadmap.
Our last benchmark post compared our tiers against each other. This one adds the comparison everyone actually asks for: the incumbent, on the same audio, against the same certified ground truth.
We also added the metrics we had been promising ourselves — the ones that score who said it, not just what was said — and ran the first measurement of a new feature that tells the engines the case's names before they guess. One of these three results is unflattering. We are publishing it anyway, because it decides what we build next.
The setup
Same corpus as before: two U.S. Supreme Court oral arguments, Nos. 25-197 and 25-466, scored against the official Heritage Reporting transcripts — 23,544 scored certified words. Public record, so anyone can obtain the same audio and ground truth we scored against. (Numbers updated 2026-08-12: scoring now reconciles orthography to the certified spelling for every contender and excludes typography and verbatim policy from both sides — see by the numbers for the method. Updated again 2026-08-13: the diarization work this post ends by promising has shipped its first piece, and the attribution table below carries the new measurements.)
New this time: we bought Rev.com's transcript of each argument and ran their export through the identical scoring path — same alignment, same normalization, same clipping to the covered region. Nothing in the scoring knows which contender is ours.
Words: every tier beats Rev, on both hearings
| Hearing | Contender | Word accuracy (higher is better) |
|---|---|---|
| 25-197 | Ours, Good | 97.5% |
| 25-197 | Ours, Advanced | 97.8% |
| 25-197 | Rev | 97.2% |
| 25-466 | Ours, Good | 96.6% |
| 25-466 | Ours, Advanced | 96.6% |
| 25-466 | Rev | 96.4% |
On 25-197 the gap is wide: our Advanced tier makes roughly 21% fewer word errors than Rev. On 25-466 it is narrower but consistent — every tier we run, including the single-engine Good tier, comes out ahead.
Names are where the difference is most visible. Rev's export of 25-466 opens by naming the petitioner "Sweepich" — the case is Sripetch v. SEC — and calls the arguing counsel, Mr. Geyser, "Geiser" throughout. On the words belonging to the case caption, Rev errs at 2.0%; our two-engine tier at 1.5%.
Who said it: the number we did not have
Word accuracy cannot see attribution. A transcript that puts every word in the right order and a third of them in the wrong mouth scores perfectly on word accuracy — and is unusable as a record.
So we added the two standard measures for it. cpWER gathers everything each speaker said and charges every word that ends up under the wrong speaker's name — and when a transcript splits one speaker in two, or merges two into one, every word in the affected turns counts against it, which is exactly how a reader experiences the mistake. WDER asks a narrower question: of the words that were transcribed, what fraction were credited to the wrong voice? Both are fractions of words affected — 0.054 means about 5.4 words per 100 — and lower is better for both.
| Hearing | Contender | cpWER | WDER |
|---|---|---|---|
| 25-197 | Ours, Good | 0.055 | 0.018 |
| 25-197 | Ours, Advanced | 0.054 | 0.020 |
| 25-197 | Ours, Extreme | 0.054 | 0.020 |
| 25-197 | Rev | 0.065 | 0.020 |
| 25-466 | Ours, Good | 0.055 | 0.013 |
| 25-466 | Ours, Advanced | 0.058 | 0.016 |
| 25-466 | Ours, Extreme | 0.060 | 0.017 |
| 25-466 | Rev | 0.054 | 0.012 |
Two things are true in that table, and we are reporting both.
On 25-197 we beat Rev at every tier. On 25-466, Rev still edges every tier of ours on attribution — 0.054 to our best 0.055 — and we publish that edge rather than average it away.
When this post first ran, the table showed a third thing: our own curve ran the wrong way — Good attributed better than Advanced, which attributed better than Extreme, on both hearings. Every engine we added made the words better and the attribution slightly worse. We said then that this pointed the next engineering cycle at diarization instead of a fourth engine, and that the work would have a baseline to beat.
That work has now shipped its first piece, and this table is its measurement. Where two voices meet — talking over each other, or one starting as the other ends — timing alone used to decide the speaker, which could interleave two speakers' sentences under one label. A language model now reads across each of those moments and re-sorts the words among the voices our speaker detection actually measured as present — it can never invent a speaker, and every word it moves still goes to a human for confirmation before anything certifies. On 25-197 that bent the curve: Advanced and Extreme now attribute better than Good on cpWER. On 25-466 the old ordering persists but every ensemble tier closed ground — Advanced from 0.062 to 0.058 against Good's 0.055. The remaining distance to Rev on that hearing is still the roadmap, and it is smaller than it was.
Telling the engines the names: honest result, modest gain
We shipped a feature this week that reads each case's caption — the heading that names the parties, the judge, the court — and hands that word list to every speech engine that can accept a "words you are likely to hear" hint before transcription starts. The theory: unusual proper names are exactly where all engines fail together, and no amount of cross-checking can catch an error every engine agrees on.
Then we measured it, on this corpus, hints on versus off (fewer errors is better):
| Hearing | Tier | Caption-word errors, off | On |
|---|---|---|---|
| 25-466 | Good | 28 | 27 |
| 25-466 | Advanced | 17 | 16 |
| 25-197 | all tiers | — | no change |
A real improvement, and a small one. The corpus explains why. A Supreme Court caption is made of words every speech engine already knows — Securities, Exchange, Commission, court, justice. The one genuinely distinctive caption word, Sripetch, is spoken once in 74 minutes of argument. And the name spoken seven times — Geyser, the one Rev misspells throughout — is an attorney's name, which appears in no caption anywhere.
Meanwhile the thing that actually fixes names on this corpus is the ensemble itself: adding the second engine cuts caption-word errors from 2.5% to 1.5% on 25-466, better than Rev's 2.0%, with no biasing involved.
So the honest reading is: the plumbing works — we verified the biased requests on the wire — but caption seeding alone is the foundation, not the payoff. The payoff comes from the two features it was built to carry: letting you upload a related court filing, which does contain counsel's names spelled correctly, and feeding every reviewer-confirmed spelling back into the case's term list so the next hearing in the same matter starts smarter. Both are on the roadmap, and now they have a metric waiting for them.
The usual caveats
This corpus is well-miked appellate argument with orderly turn-taking and clean-verbatim ground truth. It flatters everyone's attribution numbers, understates verbatim fidelity differences, and contains almost no hard names. It is a relative benchmark against a certified reference — the fairest public comparison we know how to run — not an absolute accuracy claim about your county's motion calendar. The full measurement reports behind these numbers are available on request, and the benchmark that produced them is the same one our CI runs on every release.