aharrison-fullerandClaude Opus 5
4475f6d39c
Measures median latency, strict exact-match accuracy, digit formatting and silence behaviour across 48 clips (16 Home Assistant commands x 3 macOS TTS voices), on an M4 Mac mini. Headline: Parakeet v2 at 102ms median is 5x faster than the best whisper.cpp configuration and lands within one clip of it on accuracy. mlx-whisper large-v3 is the only backend to score 48/48, at 11x the latency. Moonshine is 2x faster again but gives up real accuracy (35/48). Also quantifies the reason this defaults to v2 over v3: v3 returned digits for only 10 of 21 number-bearing commands, against 21/21 for v2, which is most of the gap between their exact-match scores. The harness feeds audio to every backend as an array rather than a path -- mlx-whisper and moonshine otherwise shell out to ffmpeg, which this project deliberately does not require. Clips are gitignored; bench/make_clips.sh regenerates them. Caveats are documented in the README: this is clean synthetic TTS, so it measures latency rigorously and accuracy only as a domain smoke test, and faster-whisper is CPU-only on Apple Silicon because CTranslate2 has no Metal backend. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
36 lines
1.1 KiB
Bash
Executable File
36 lines
1.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Render the benchmark corpus to 16kHz mono WAV using macOS TTS, across
|
|
# several voices. Synthetic speech is clean and accent-consistent, so treat
|
|
# the accuracy numbers as a domain smoke test, not a WER benchmark.
|
|
set -euo pipefail
|
|
|
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
OUT="$DIR/clips"
|
|
VOICES=(Daniel Samantha Karen)
|
|
|
|
rm -rf "$OUT"; mkdir -p "$OUT"
|
|
|
|
i=0
|
|
while IFS=$'\t' read -r phrase _expected _has_number; do
|
|
[[ "$phrase" =~ ^# || -z "$phrase" ]] && continue
|
|
i=$((i + 1))
|
|
for voice in "${VOICES[@]}"; do
|
|
if ! say -v "$voice" -o "$OUT/${i}_${voice}.aiff" "$phrase" 2>/dev/null; then
|
|
say -o "$OUT/${i}_${voice}.aiff" "$phrase"
|
|
fi
|
|
afconvert -f WAVE -d LEI16@16000 -c 1 \
|
|
"$OUT/${i}_${voice}.aiff" "$OUT/${i}_${voice}.wav"
|
|
rm -f "$OUT/${i}_${voice}.aiff"
|
|
done
|
|
done < "$DIR/corpus.tsv"
|
|
|
|
python3 - "$OUT/silence.wav" <<'PY'
|
|
import sys, wave
|
|
w = wave.open(sys.argv[1], "wb")
|
|
w.setparams((1, 2, 16000, 0, "NONE", "NONE"))
|
|
w.writeframes(b"\x00\x00" * 16000 * 3)
|
|
w.close()
|
|
PY
|
|
|
|
echo "Wrote $(ls "$OUT"/*.wav | wc -l | tr -d ' ') clips to $OUT"
|