Benchmarks & methodology

Zephyr, our speech-to-text engine, is evaluated with a fully reproducible, provider-agnostic suite: every benchmark is generated from public datasets using deterministic, seeded scripts, and produces identical outputs regardless of provider. We publish the harness so you can verify the numbers on your own — nothing here is hand-picked.

What we measure

Six benchmarks, each from public data:

Reproduce it yourself

The suite is the SpeechRevolutions/benchmarks repository. It talks to the production API through the published speechrevolutions SDK — there is nothing to host and no local stack to run. Point it at any competitor with that provider's API key instead; every provider returns the same normalized transcript, so scoring is identical.

git clone https://github.com/SpeechRevolutions/benchmarks
cd benchmarks

pip install -r requirements.txt
python -m spacy download en_core_web_sm          # entity benchmark

# 1. Build the frozen datasets from public sources (deterministic).
#    The audio is not committed, so this step is required.
python -m benchmarks.datasets.prepare_all

# 2. Run one benchmark, or all of them, against our production API
export SPEECHREVOLUTIONS_API_KEY=stt_...
python -m benchmarks.cli run wer
python -m benchmarks.cli run all

# 3. Run against another provider (needs that provider's API key)
DEEPGRAM_API_KEY=... python -m benchmarks.cli run all --provider deepgram
ASSEMBLYAI_API_KEY=... python -m benchmarks.cli run all --provider assemblyai

python -m benchmarks.cli list shows every benchmark and provider. Results are written as JSON / Markdown / CSV. Every command above runs from the repository root.

Methodology we hold ourselves to

• WER uses the standard Whisper text normalizers; diarization DER is overlap-aware and scored at the same 0.25 s collar for every provider, with strict collar-0 figures reported alongside.
• Where a metric requires data a provider can't emit (e.g. per-word language labels for switch latency, or word timestamps from a text-only API), it is reported as null — never fabricated.
• Custom-vocabulary glossaries, when used, are applied identically to every keyword-capable provider through its own native parameter, so comparisons stay fair.