aharrison-fullerandClaude Opus 5
4475f6d39c
Measures median latency, strict exact-match accuracy, digit formatting and silence behaviour across 48 clips (16 Home Assistant commands x 3 macOS TTS voices), on an M4 Mac mini. Headline: Parakeet v2 at 102ms median is 5x faster than the best whisper.cpp configuration and lands within one clip of it on accuracy. mlx-whisper large-v3 is the only backend to score 48/48, at 11x the latency. Moonshine is 2x faster again but gives up real accuracy (35/48). Also quantifies the reason this defaults to v2 over v3: v3 returned digits for only 10 of 21 number-bearing commands, against 21/21 for v2, which is most of the gap between their exact-match scores. The harness feeds audio to every backend as an array rather than a path -- mlx-whisper and moonshine otherwise shell out to ffmpeg, which this project deliberately does not require. Clips are gitignored; bench/make_clips.sh regenerates them. Caveats are documented in the README: this is clean synthetic TTS, so it measures latency rigorously and accuracy only as a domain smoke test, and faster-whisper is CPU-only on Apple Silicon because CTranslate2 has no Metal backend. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1.3 KiB
1.3 KiB
| 1 | # tts_input | expected_transcript | has_number |
|---|---|---|---|
| 2 | turn off the kitchen lights | turn off the kitchen lights | 0 |
| 3 | set the living room thermostat to twenty one degrees | set the living room thermostat to 21 degrees | 1 |
| 4 | what is the temperature in the bedroom | what is the temperature in the bedroom | 0 |
| 5 | dim the hallway lights to thirty percent | dim the hallway lights to 30% | 1 |
| 6 | is the back door locked | is the back door locked | 0 |
| 7 | turn on the christmas tree in the conservatory | turn on the christmas tree in the conservatory | 0 |
| 8 | set a timer for twelve minutes | set a timer for 12 minutes | 1 |
| 9 | play radio six music in the kitchen | play radio 6 music in the kitchen | 1 |
| 10 | what's the octopus agile rate right now | what's the octopus agile rate right now | 0 |
| 11 | close the blinds in the study and turn on the desk lamp | close the blinds in the study and turn on the desk lamp | 0 |
| 12 | turn the bedroom lights down to five percent | turn the bedroom lights down to 5% | 1 |
| 13 | how much solar am I generating | how much solar am i generating | 0 |
| 14 | lock the front door and turn off all the lights | lock the front door and turn off all the lights | 0 |
| 15 | set the upstairs aircon to eighteen degrees | set the upstairs aircon to 18 degrees | 1 |
| 16 | pause the music in the living room | pause the music in the living room | 0 |
| 17 | remind me in forty five minutes | remind me in 45 minutes | 1 |