aharrison-fuller/wyoming-parakeet-mlx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add benchmark harness and compare against 11 other STT backends

Browse Source 
Measures median latency, strict exact-match accuracy, digit formatting and
silence behaviour across 48 clips (16 Home Assistant commands x 3 macOS TTS
voices), on an M4 Mac mini.

Headline: Parakeet v2 at 102ms median is 5x faster than the best whisper.cpp
configuration and lands within one clip of it on accuracy. mlx-whisper
large-v3 is the only backend to score 48/48, at 11x the latency. Moonshine is
2x faster again but gives up real accuracy (35/48).

Also quantifies the reason this defaults to v2 over v3: v3 returned digits
for only 10 of 21 number-bearing commands, against 21/21 for v2, which is
most of the gap between their exact-match scores.

The harness feeds audio to every backend as an array rather than a path --
mlx-whisper and moonshine otherwise shell out to ffmpeg, which this project
deliberately does not require.

Clips are gitignored; bench/make_clips.sh regenerates them. Caveats are
documented in the README: this is clean synthetic TTS, so it measures latency
rigorously and accuracy only as a domain smoke test, and faster-whisper is
CPU-only on Apple Silicon because CTranslate2 has no Metal backend.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
aharrison-fuller
2026-07-29 03:32:57 +01:00
co-authored by Claude Opus 5
parent c846bc245c
commit 4475f6d39c
6 changed files with 373 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bench/corpus.tsv
+17
View File
@@ -0,0 +1,17 @@
# tts_input expected_transcript has_number
turn off the kitchen lights turn off the kitchen lights 0
set the living room thermostat to twenty one degrees set the living room thermostat to 21 degrees 1
what is the temperature in the bedroom what is the temperature in the bedroom 0
dim the hallway lights to thirty percent dim the hallway lights to 30% 1
is the back door locked is the back door locked 0
turn on the christmas tree in the conservatory turn on the christmas tree in the conservatory 0
set a timer for twelve minutes set a timer for 12 minutes 1
play radio six music in the kitchen play radio 6 music in the kitchen 1
what's the octopus agile rate right now what's the octopus agile rate right now 0
close the blinds in the study and turn on the desk lamp close the blinds in the study and turn on the desk lamp 0
turn the bedroom lights down to five percent turn the bedroom lights down to 5% 1
how much solar am I generating how much solar am i generating 0
lock the front door and turn off all the lights lock the front door and turn off all the lights 0
set the upstairs aircon to eighteen degrees set the upstairs aircon to 18 degrees 1
pause the music in the living room pause the music in the living room 0
remind me in forty five minutes remind me in 45 minutes 1
1 # tts_input expected_transcript has_number
2 turn off the kitchen lights turn off the kitchen lights 0
3 set the living room thermostat to twenty one degrees set the living room thermostat to 21 degrees 1
4 what is the temperature in the bedroom what is the temperature in the bedroom 0
5 dim the hallway lights to thirty percent dim the hallway lights to 30% 1
6 is the back door locked is the back door locked 0
7 turn on the christmas tree in the conservatory turn on the christmas tree in the conservatory 0
8 set a timer for twelve minutes set a timer for 12 minutes 1
9 play radio six music in the kitchen play radio 6 music in the kitchen 1
10 what's the octopus agile rate right now what's the octopus agile rate right now 0
11 close the blinds in the study and turn on the desk lamp close the blinds in the study and turn on the desk lamp 0
12 turn the bedroom lights down to five percent turn the bedroom lights down to 5% 1
13 how much solar am I generating how much solar am i generating 0
14 lock the front door and turn off all the lights lock the front door and turn off all the lights 0
15 set the upstairs aircon to eighteen degrees set the upstairs aircon to 18 degrees 1
16 pause the music in the living room pause the music in the living room 0
17 remind me in forty five minutes remind me in 45 minutes 1