Benchmarks & methodology

Zephyr, the Speech Revolutions speech-to-text engine, is evaluated with a reproducible, provider-agnostic suite. Every benchmark is generated from public datasets with deterministic, seeded scripts and is scored the same way for every provider. The harness is public, so you can verify the numbers yourself.

Headline results

Measured by Speech Revolutions on public datasets, last run 2026-07-12/13 against each provider's then-current model. Lower is better for every metric.

Diarization figures are overlap-aware and scored at a 0.25 s collar. This is the lenient condition and the one quoted throughout this site. The same run also reports strict collar-0 figures, which is the condition the DiariZen and pyannote model cards use; compare those against a published model card. The suite prints both, overall and per dataset.

What is measured

Six benchmarks, each from public data:

Reproduce it yourself

The suite is the SpeechRevolutions/benchmarks repository. It calls the production API through the published speechrevolutions SDK, so there is nothing to host. To benchmark another provider, supply that provider's API key. Every provider's output is normalized to the same transcript shape, so scoring is identical.

git clone https://github.com/SpeechRevolutions/benchmarks
cd benchmarks

pip install -r requirements.txt
python -m spacy download en_core_web_sm          # entity benchmark

# 1. Build the frozen datasets from public sources (deterministic).
#    The audio is not committed, so this step is required.
python -m benchmarks.datasets.prepare_all

# 2. Run one benchmark, or all of them, against our production API
export SPEECHREVOLUTIONS_API_KEY=stt_...
python -m benchmarks.cli run wer
python -m benchmarks.cli run all

# 3. Run against another provider (needs that provider's API key)
DEEPGRAM_API_KEY=... python -m benchmarks.cli run all --provider deepgram
ASSEMBLYAI_API_KEY=... python -m benchmarks.cli run all --provider assemblyai

python -m benchmarks.cli list shows every benchmark and provider. Results are written as JSON, Markdown, and CSV. Run every command above from the repository root.

Methodology

• WER uses the standard Whisper text normalizers. Diarization DER is overlap-aware and scored at the same 0.25 s collar for every provider, with strict collar-0 figures reported alongside.
• When a metric needs data a provider doesn't return (for example, per-word language labels for switch latency, or word timestamps from a text-only API), it is reported as null, never estimated.
• Custom-vocabulary glossaries, when used, are applied identically to every provider that supports keywords, through that provider's native parameter.