AI Transcription Accuracy Test Kit: WER Is Only the First Metric

A transcription comparison is only fair when every system receives the same audio, settings are recorded, and output is compared with a human-corrected reference. A vendor demo, confidence number, or clean podcast clip cannot answer whether a system handles your speakers, names, noise, languages, captions, and privacy requirements.

Word error rate is a useful base metric, but it treats every word similarly. For meeting actions, a wrong date or owner may matter more than three filler-word errors. For captions, timing and meaningful non-speech audio matter. This kit combines WER with job-specific measures.

Define the finished job

Choose one primary use: searchable meeting notes, call analytics, interview editing, subtitles, live captions, legal review, medical documentation, or application speech input. Record whether output must be verbatim or cleaned, whether speaker labels and timestamps are required, which languages and accents occur, acceptable delay, human-review level, retention, and error severity.

Set a release rule before testing. Example: no wrong account numbers, dates, or consent statements; WER below the team's baseline on every priority stratum; at least 95 percent correct speaker attribution for two-person calls; and median correction time under eight minutes per 30 minutes of audio. Replace those example thresholds with requirements validated for the actual use.

Build a representative audio matrix

Use recordings you are authorized to process, preferably scripted or consented evaluation material. Include at least 20 clips and stratify them:

Quiet close microphone and ordinary laptop microphone.
Background noise, compression, packet loss, and distant speech.
One speaker, two speakers, overlap, interruption, and rapid turn-taking.
Every supported language, accent, and code-switching pattern used in production.
Names, acronyms, addresses, currency, dates, measurements, and domain terms.
Short replies, silence, laughter, music, alarms, and other meaningful sounds when captions are the output.

Keep duration reasonably balanced across strata. One long clean recording should not drown out five difficult but important clips in the aggregate score.

Create a defensible ground truth

Have two trained reviewers transcribe each clip independently, then adjudicate differences while listening to the source. Preserve speaker turns, critical entities, and uncertainty. Store an unnormalized reference and a normalized copy used for WER. The ground truth is itself fallible, so record reviewer and adjudicator IDs.

Freeze normalization rules before scoring. Decide how to handle punctuation, capitalization, contractions, filler words, hesitations, numerals, hyphenation, abbreviations, and non-speech events. Apply the same rules to every provider. Publish the rules with the result; otherwise the WER is not reproducible.

Calculate WER correctly

Google's speech-accuracy guide describes word error rate as the standard comparison metric and defines substitution, deletion, and insertion errors against a human ground truth. The formula is:

WER = (substitutions + deletions + insertions) / reference words

Report the numerator counts and reference length, not only a percentage. Calculate per clip and per stratum before the aggregate. For streaming systems, Deepgram recommends concatenating final transcripts for WER rather than scoring interim text, which is expected to change as more audio arrives.

Add metrics that match the operational risk

Metric	How to score	Why it matters
Critical entity error rate	Wrong or missing critical entities / reference entities	Names, dates, amounts, identifiers can drive action
Speaker attribution	Correctly attributed words or turns / scored words or turns	Wrong speaker can reverse responsibility
Timestamp error	Absolute offset from reference boundaries	Search, subtitles, and editing depend on alignment
Caption completeness	Dialogue and meaningful audio events represented	Access requires more than readable prose
Correction time	Minutes from raw transcript to approved output	Captures punctuation, formatting, and workflow cost
Failure rate	Timeouts, empty results, truncated jobs / attempts	A missing transcript is not an accuracy score

W3C's accessibility checks explain that audio information needs an alternative such as captions or a transcript for people who cannot hear it. If accessibility is the goal, involve users with disabilities and test the complete media player and caption workflow, not just word recognition.

Do not compare provider confidence scores directly

Deepgram's documentation distinguishes word-level and transcript-level confidence and explicitly warns that confidence definitions differ across providers and are not a WER guarantee. A model can be confidently wrong, especially on domain terms or mismatched audio. Use confidence within a provider to route uncertain words for review only after measuring precision and recall against your own labeled data.

For critical entities, a median transcript confidence can hide one very low-confidence account number. Inspect entity-level confidence where available and maintain a deterministic validator for formats such as dates, amounts, and identifiers.

Test adaptation without contaminating the comparison

Run a default pass first. Then create a documented domain vocabulary and run an adapted pass where each product supports it. Google documents phrase sets, custom classes, and optional boost for helping recognition of frequent, rare, or domain-specific terms. Report default and adapted results separately; adaptation is part of the production configuration and has maintenance cost.

Split audio into development and holdout sets. Tune vocabulary or thresholds on the development set, then report the holdout result. Repeatedly adjusting settings on the same evaluation clips overstates expected performance.

Measure privacy and lifecycle beside accuracy

Record upload path, processing region, human access, training use, abuse-monitoring logs, transcript and audio retention, deletion, export, administrator controls, and subcontractors for the exact plan. Otter documents a custom data-retention policy, but applicability must be confirmed for the account. A more accurate service can still be unacceptable if the data lifecycle conflicts with consent, records policy, or contract.

Use synthetic sensitive data in the pilot. Verify deletion through the product and administrator path. Keep the evaluation audio only as long as the test policy permits.

Run and report the comparison

Freeze audio, reference, normalization, plan, model, language, and options.
Submit the same original files; avoid re-encoding for one provider only.
Preserve raw responses, request IDs, errors, and final transcripts.
Score blindly where possible and adjudicate critical errors.
Report each stratum, confidence intervals where the sample supports them, and every hard failure.
Document the approved scope and re-test trigger.

Do not publish a universal winner from one language or microphone. The useful result is conditional: which configuration meets the defined thresholds for which audio and workflow.

Original worksheet

Download the transcription accuracy test log (CSV). It includes audio strata, normalization version, substitutions, deletions, insertions, reference words, critical entities, speakers, timing, correction minutes, failures, retention checks, and release decision.

Source map

Google's speech-accuracy guide supports the ground-truth and WER definitions.
Google's adaptation guide supports the phrase-set, class, and boost discussion.
Deepgram's confidence guide supports the warning against treating confidence as WER or comparing raw scores across providers; its interim-results guide supports scoring concatenated final output.
Otter's retention article supports the existence of configurable retention on applicable accounts.
W3C WAI's accessibility checks support the need for captions or transcripts as media alternatives.

Limitations

This kit was source-reviewed on July 17, 2026. It does not independently benchmark any provider. Twenty clips are a pilot, not proof across all speakers, disabilities, languages, devices, or environments. WER normalization can materially change results, and caption quality requires human accessibility review. Product models, languages, confidence calibration, retention, and pricing can change. High-stakes transcription requires qualified human verification and applicable legal and compliance review.