Measures median latency, strict exact-match accuracy, digit formatting and
silence behaviour across 48 clips (16 Home Assistant commands x 3 macOS TTS
voices), on an M4 Mac mini.
Headline: Parakeet v2 at 102ms median is 5x faster than the best whisper.cpp
configuration and lands within one clip of it on accuracy. mlx-whisper
large-v3 is the only backend to score 48/48, at 11x the latency. Moonshine is
2x faster again but gives up real accuracy (35/48).
Also quantifies the reason this defaults to v2 over v3: v3 returned digits
for only 10 of 21 number-bearing commands, against 21/21 for v2, which is
most of the gap between their exact-match scores.
The harness feeds audio to every backend as an array rather than a path --
mlx-whisper and moonshine otherwise shell out to ffmpeg, which this project
deliberately does not require.
Clips are gitignored; bench/make_clips.sh regenerates them. Caveats are
documented in the README: this is clean synthetic TTS, so it measures latency
rigorously and accuracy only as a domain smoke test, and faster-whisper is
CPU-only on Apple Silicon because CTranslate2 has no Metal backend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
aharrison-fullerandClaude Opus 5
4475f6d39c