From 4475f6d39cb97acc0c7964a82e38d96e8152660e Mon Sep 17 00:00:00 2001 From: Adam Harrison-Fuller Date: Wed, 29 Jul 2026 03:32:57 +0100 Subject: [PATCH] Add benchmark harness and compare against 11 other STT backends Measures median latency, strict exact-match accuracy, digit formatting and silence behaviour across 48 clips (16 Home Assistant commands x 3 macOS TTS voices), on an M4 Mac mini. Headline: Parakeet v2 at 102ms median is 5x faster than the best whisper.cpp configuration and lands within one clip of it on accuracy. mlx-whisper large-v3 is the only backend to score 48/48, at 11x the latency. Moonshine is 2x faster again but gives up real accuracy (35/48). Also quantifies the reason this defaults to v2 over v3: v3 returned digits for only 10 of 21 number-bearing commands, against 21/21 for v2, which is most of the gap between their exact-match scores. The harness feeds audio to every backend as an array rather than a path -- mlx-whisper and moonshine otherwise shell out to ffmpeg, which this project deliberately does not require. Clips are gitignored; bench/make_clips.sh regenerates them. Caveats are documented in the README: this is clean synthetic TTS, so it measures latency rigorously and accuracy only as a domain smoke test, and faster-whisper is CPU-only on Apple Silicon because CTranslate2 has no Metal backend. Co-Authored-By: Claude Opus 5 --- .gitignore | 2 + README.md | 89 +++++++++++++--- bench/benchmark.py | 235 +++++++++++++++++++++++++++++++++++++++++ bench/corpus.tsv | 17 +++ bench/make_clips.sh | 35 ++++++ bench/requirements.txt | 7 ++ 6 files changed, 373 insertions(+), 12 deletions(-) create mode 100755 bench/benchmark.py create mode 100644 bench/corpus.tsv create mode 100755 bench/make_clips.sh create mode 100644 bench/requirements.txt diff --git a/.gitignore b/.gitignore index cfa63b4..4865f5b 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,5 @@ __pycache__/ .pytest_cache/ # Generated by test/make-clips.sh test/clips/ +# Generated by bench/make_clips.sh +bench/clips/ diff --git a/README.md b/README.md index 563808d..432c0aa 100644 --- a/README.md +++ b/README.md @@ -10,23 +10,84 @@ bridge and inference. ## Why -Measured on an M4 Mac mini against ten typical Home Assistant voice commands, -replacing a whisper.cpp setup: - -| Backend | Mean latency | Correct | Silent input | -|---|---|---|---| -| whisper.cpp `large-v3` | ~1150 ms | 10/10 | `"Thank you."` | -| whisper.cpp `large-v3-turbo` | ~570 ms | 10/10 | `"Thank you."` | -| **parakeet-tdt-0.6b-v2** | **~110 ms** | **10/10** | `""` | +On an M4 Mac mini, Parakeet transcribes a typical Home Assistant command in +**~100 ms** — roughly 5× faster than the best whisper.cpp configuration and +9× faster than whisper `large-v3`, while matching them on accuracy. See +[Benchmarks](#benchmarks) for the full comparison against eleven other +backends. Two things matter beyond raw speed: -- **Silence returns an empty string.** Whisper hallucinates `"Thank you."` on - digital silence, which reaches your conversation agent as a real utterance. +- **Silence returns an empty string.** Every whisper variant tested + hallucinates on digital silence (`"Thank you."`, or `"you"` for + faster-whisper), which reaches your conversation agent as a real utterance. + Parakeet and Moonshine return nothing. - **No cross-request contamination.** whisper.cpp's server carries decoder context between requests unless you pass `-nc`, and will return the *previous* utterance — in testing, roughly one time in five. +## Benchmarks + +All figures measured on one machine: **M4 Mac mini (10-core, 32 GB), macOS 26.5**. +48 clips — 16 Home Assistant commands rendered through three macOS TTS voices +(Daniel, Samantha, Karen). Every backend loads once, transcribes all 48 clips +to warm up, then runs a timed pass. Latency is the **median** of that pass; +means are skewed by first-request kernel compilation. + +Reproduce with `./bench/make_clips.sh && ./bench/benchmark.py --backend ...`. + +| Backend | Runtime | Median | p90 | Exact | Digits | Silence | +|---|---|--:|--:|--:|--:|---| +| moonshine tiny | ONNX CPU | 28 ms | 44 ms | 31/48 | 15/21 | `""` | +| moonshine base | ONNX CPU | 53 ms | 70 ms | 35/48 | 20/21 | `""` | +| **parakeet-tdt-0.6b-v2** | **MLX** | **102 ms** | **112 ms** | **46/48** | **21/21** | `""` | +| parakeet-tdt-0.6b-v3 | MLX | 128 ms | 149 ms | 36/48 | 10/21 | `""` | +| faster-whisper tiny.en | CPU int8 | 182 ms | 204 ms | 44/48 | 21/21 | `"you"` | +| faster-whisper base.en | CPU int8 | 326 ms | 347 ms | 44/48 | 19/21 | `"you"` | +| whisper.cpp large-v3-turbo | Metal + CoreML | 518 ms | 537 ms | 47/48 | 21/21 | `"thank you"` | +| mlx-whisper large-v3-turbo | MLX | 828 ms | 848 ms | 47/48 | 21/21 | `"thank you"` | +| whisper.cpp large-v3 | Metal + CoreML | 941 ms | 1042 ms | 47/48 | 21/21 | `"thank you"` | +| faster-whisper small.en | CPU int8 | 974 ms | 1032 ms | 47/48 | 21/21 | `"you"` | +| mlx-whisper large-v3 | MLX | 1170 ms | 1251 ms | 48/48 | 21/21 | `"thank you"` | +| faster-whisper distil-large-v3 | CPU int8 | 4269 ms | 4304 ms | 46/48 | 21/21 | `"thank you"` | + +**Exact** is a strict string match after normalising case, punctuation and +whitespace. **Digits** counts how many of the 21 number-bearing clips came +back with digits rather than spelled-out words — see +[Model choice](#model-choice-v2-not-v3) for why that matters more than it looks. +**Silence** is the output for three seconds of digital silence. + +What the numbers say: + +- **Parakeet v2 has the best latency/accuracy trade-off here.** It is 5× + faster than the best whisper.cpp configuration and lands within one clip of + it on accuracy. +- **`mlx-whisper large-v3` is the accuracy ceiling** — the only backend to + score 48/48 — but costs 11× the latency to get there. +- **Moonshine is genuinely faster**, at 2× Parakeet's speed, and it also + handles silence cleanly. It gives up real accuracy for it (35/48), so it is + the right pick only if latency dominates everything else. +- **Parakeet v3's 10/21 on digits** is the ITN problem quantified. Its exact + match (36/48) is dragged down almost entirely by that one behaviour. +- Both clips Parakeet v2 misses are the same word — "aircon", which the TTS + voices render as "air con" / "aircan". Accuracy differences at the top of + this table are concentrated in a couple of awkward tokens, not spread out. + +### Caveats — read these before trusting the table + +- **This is not a WER benchmark.** The audio is clean synthetic TTS from three + similar English voices, with no noise, accents, crosstalk or far-field + effects. It measures latency rigorously and accuracy only as a domain smoke + test. For real word error rates see the + [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard). +- **faster-whisper is CPU-only on Apple Silicon.** CTranslate2 has no Metal + backend, so those rows show CPU int8 performance. On an NVIDIA GPU they + would look completely different — do not read this as a verdict on + faster-whisper generally, only on what it does on this hardware. +- **Latency is raw inference**, excluding Wyoming protocol overhead. End to + end through this server, expect roughly 15–35 ms on top. +- One machine, one run each. Treat differences of a few percent as noise. + ## Requirements - Apple Silicon Mac (MLX is Metal/ANE-backed) @@ -73,8 +134,12 @@ multilingual, because of inverse text normalisation: Home Assistant's local intent matching (hassil) expects digits. If your pipeline has `prefer_local_intents` enabled, a model that spells numbers out still *looks* accurate while quietly pushing commands off the fast local path -onto your LLM fallback. v3 is the better choice if you need languages other -than English — just be aware of the trade. +onto your LLM fallback. + +Measured on the benchmark corpus, v3 returned digits for only **10 of 21** +number-bearing commands, against **21/21** for v2. That single behaviour is +most of the gap between their exact-match scores. v3 remains the right choice +if you need languages other than English — just size the trade-off first. ## Updating the model diff --git a/bench/benchmark.py b/bench/benchmark.py new file mode 100755 index 0000000..1080f4b --- /dev/null +++ b/bench/benchmark.py @@ -0,0 +1,235 @@ +#!/usr/bin/env python3 +"""Benchmark speech-to-text backends on the Home Assistant command corpus. + +Each backend is loaded once, warmed on every clip, then timed. Reported +latency is the median of the timed pass, which is what a voice assistant +actually experiences -- means are skewed by Metal kernel compilation on the +first request. + +Usage: + ./benchmark.py --backend parakeet:mlx-community/parakeet-tdt-0.6b-v2 + ./benchmark.py --backend mlx-whisper:mlx-community/whisper-large-v3-turbo + ./benchmark.py --backend faster-whisper:base.en + ./benchmark.py --backend moonshine:moonshine/base + ./benchmark.py --backend whispercpp:http://127.0.0.1:8910/inference +""" +import argparse +import json +import re +import statistics +import sys +import time +import unicodedata +from pathlib import Path + +BENCH_DIR = Path(__file__).resolve().parent +CLIPS = BENCH_DIR / "clips" + +# Spoken-form numbers a model might emit instead of digits. Home Assistant's +# local intent matching wants digits, so we score this separately. +WORD_NUMBERS = re.compile( + r"\b(one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve|" + r"thirteen|fourteen|fifteen|sixteen|seventeen|eighteen|nineteen|twenty|" + r"thirty|forty|fifty|sixty|seventy|eighty|ninety|hundred|percent)\b" +) + + +def normalise(text: str) -> str: + """Fold away differences that do not change the intent: case, smart + quotes, punctuation, and whitespace. '%' is kept -- it is semantic.""" + text = unicodedata.normalize("NFKD", text).lower().strip() + text = text.replace("’", "'").replace("‘", "'") + text = re.sub(r"[^\w\s%']", " ", text) + return re.sub(r"\s+", " ", text).strip() + + +def load_corpus(): + rows = [] + for line in (BENCH_DIR / "corpus.tsv").read_text().splitlines(): + if not line.strip() or line.startswith("#"): + continue + _tts, expected, has_number = line.split("\t") + rows.append((expected, has_number == "1")) + return rows + + +# --- backends --------------------------------------------------------------- + + +def read_wav(path): + """Read a 16kHz mono 16-bit WAV as float32 in [-1, 1). + + Several of these libraries shell out to ffmpeg to load audio, which is an + unnecessary dependency when the clips are already in the right format -- + and unavailable on the benchmark machine. Feed them arrays instead. + """ + import wave + + import numpy as np + + with wave.open(str(path), "rb") as w: + assert w.getframerate() == 16000 and w.getnchannels() == 1 + pcm = w.readframes(w.getnframes()) + return np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0 + + +def backend_parakeet(model_id): + import mlx.core as mx + from parakeet_mlx import from_pretrained + from parakeet_mlx.audio import get_logmel + + model = from_pretrained(model_id) + + def transcribe(path): + mel = get_logmel(mx.array(read_wav(path)), model.preprocessor_config) + results = model.generate(mel) + return results[0].text if results else "" + + return transcribe + + +def backend_mlx_whisper(model_id): + import mlx_whisper + + def transcribe(path): + return mlx_whisper.transcribe( + read_wav(path), path_or_hf_repo=model_id, language="en", fp16=True + )["text"] + + return transcribe + + +def backend_faster_whisper(model_id): + from faster_whisper import WhisperModel + + # Metal is unsupported by CTranslate2; int8 on CPU is the fastest option + # available on Apple Silicon and is what the HA add-on uses by default. + model = WhisperModel(model_id, device="cpu", compute_type="int8") + + def transcribe(path): + segments, _info = model.transcribe(str(path), language="en", beam_size=5) + return "".join(s.text for s in segments) + + return transcribe + + +def backend_moonshine(model_id): + import moonshine_onnx + + model = moonshine_onnx.MoonshineOnnxModel(model_name=model_id) + tokenizer = moonshine_onnx.load_tokenizer() + + def transcribe(path): + # Bypass moonshine_onnx.transcribe() so we can supply the audio as an + # array; it expects shape [batch, samples]. + audio = read_wav(path).reshape(1, -1) + return " ".join(tokenizer.decode_batch(model.generate(audio))) + + return transcribe + + +def backend_whispercpp(url): + import requests + + def transcribe(path): + with open(path, "rb") as fh: + r = requests.post( + url, + files={"file": fh}, + data={"response_format": "json", "no_context": "true"}, + timeout=120, + ) + r.raise_for_status() + return r.json()["text"] + + return transcribe + + +BACKENDS = { + "parakeet": backend_parakeet, + "mlx-whisper": backend_mlx_whisper, + "faster-whisper": backend_faster_whisper, + "moonshine": backend_moonshine, + "whispercpp": backend_whispercpp, +} + + +# --- runner ----------------------------------------------------------------- + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--backend", required=True, help="kind:model_or_url") + ap.add_argument("--label", help="Name to report (defaults to --backend)") + ap.add_argument("--json", action="store_true", help="Emit a JSON result line") + args = ap.parse_args() + + kind, _, target = args.backend.partition(":") + if kind not in BACKENDS: + sys.exit(f"unknown backend {kind!r}; pick one of {', '.join(BACKENDS)}") + + corpus = load_corpus() + clips = sorted( + (p for p in CLIPS.glob("*.wav") if p.name != "silence.wav"), + key=lambda p: (int(p.stem.split("_")[0]), p.stem), + ) + if not clips: + sys.exit("no clips found -- run ./make_clips.sh first") + + load_started = time.monotonic() + transcribe = BACKENDS[kind](target) + load_seconds = time.monotonic() - load_started + + # Warm every clip first: the first inference compiles kernels, and clip + # length varies enough that a single warm-up does not cover all shapes. + for clip in clips: + transcribe(clip) + + latencies, exact, number_ok, number_total, failures = [], 0, 0, 0, [] + for clip in clips: + index = int(clip.stem.split("_")[0]) - 1 + expected, has_number = corpus[index] + + started = time.monotonic() + got = transcribe(clip) + latencies.append((time.monotonic() - started) * 1000) + + if normalise(got) == normalise(expected): + exact += 1 + else: + failures.append((clip.name, normalise(expected), normalise(got))) + if has_number: + number_total += 1 + if not WORD_NUMBERS.search(normalise(got)): + number_ok += 1 + + silence = CLIPS / "silence.wav" + silence_out = normalise(transcribe(silence)) if silence.exists() else "n/a" + + label = args.label or args.backend + result = { + "backend": label, + "clips": len(clips), + "median_ms": round(statistics.median(latencies), 1), + "p90_ms": round(sorted(latencies)[int(len(latencies) * 0.9)], 1), + "exact_match": f"{exact}/{len(clips)}", + "digits_ok": f"{number_ok}/{number_total}", + "silence": silence_out, + "load_s": round(load_seconds, 1), + } + + if args.json: + print(json.dumps(result)) + else: + print(f"\n=== {label} ===") + for key, value in result.items(): + if key != "backend": + print(f" {key:12s} {value}") + if failures: + print(f" mismatches ({len(failures)}):") + for name, want, got in failures[:8]: + print(f" {name}\n want: {want}\n got: {got}") + + +if __name__ == "__main__": + main() diff --git a/bench/corpus.tsv b/bench/corpus.tsv new file mode 100644 index 0000000..dab86c7 --- /dev/null +++ b/bench/corpus.tsv @@ -0,0 +1,17 @@ +# tts_input expected_transcript has_number +turn off the kitchen lights turn off the kitchen lights 0 +set the living room thermostat to twenty one degrees set the living room thermostat to 21 degrees 1 +what is the temperature in the bedroom what is the temperature in the bedroom 0 +dim the hallway lights to thirty percent dim the hallway lights to 30% 1 +is the back door locked is the back door locked 0 +turn on the christmas tree in the conservatory turn on the christmas tree in the conservatory 0 +set a timer for twelve minutes set a timer for 12 minutes 1 +play radio six music in the kitchen play radio 6 music in the kitchen 1 +what's the octopus agile rate right now what's the octopus agile rate right now 0 +close the blinds in the study and turn on the desk lamp close the blinds in the study and turn on the desk lamp 0 +turn the bedroom lights down to five percent turn the bedroom lights down to 5% 1 +how much solar am I generating how much solar am i generating 0 +lock the front door and turn off all the lights lock the front door and turn off all the lights 0 +set the upstairs aircon to eighteen degrees set the upstairs aircon to 18 degrees 1 +pause the music in the living room pause the music in the living room 0 +remind me in forty five minutes remind me in 45 minutes 1 diff --git a/bench/make_clips.sh b/bench/make_clips.sh new file mode 100755 index 0000000..7a3118c --- /dev/null +++ b/bench/make_clips.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Render the benchmark corpus to 16kHz mono WAV using macOS TTS, across +# several voices. Synthetic speech is clean and accent-consistent, so treat +# the accuracy numbers as a domain smoke test, not a WER benchmark. +set -euo pipefail + +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +OUT="$DIR/clips" +VOICES=(Daniel Samantha Karen) + +rm -rf "$OUT"; mkdir -p "$OUT" + +i=0 +while IFS=$'\t' read -r phrase _expected _has_number; do + [[ "$phrase" =~ ^# || -z "$phrase" ]] && continue + i=$((i + 1)) + for voice in "${VOICES[@]}"; do + if ! say -v "$voice" -o "$OUT/${i}_${voice}.aiff" "$phrase" 2>/dev/null; then + say -o "$OUT/${i}_${voice}.aiff" "$phrase" + fi + afconvert -f WAVE -d LEI16@16000 -c 1 \ + "$OUT/${i}_${voice}.aiff" "$OUT/${i}_${voice}.wav" + rm -f "$OUT/${i}_${voice}.aiff" + done +done < "$DIR/corpus.tsv" + +python3 - "$OUT/silence.wav" <<'PY' +import sys, wave +w = wave.open(sys.argv[1], "wb") +w.setparams((1, 2, 16000, 0, "NONE", "NONE")) +w.writeframes(b"\x00\x00" * 16000 * 3) +w.close() +PY + +echo "Wrote $(ls "$OUT"/*.wav | wc -l | tr -d ' ') clips to $OUT" diff --git a/bench/requirements.txt b/bench/requirements.txt new file mode 100644 index 0000000..d902ee6 --- /dev/null +++ b/bench/requirements.txt @@ -0,0 +1,7 @@ +# Backends compared in the README benchmark table. Install into a throwaway +# venv -- not the service venv. +parakeet-mlx +mlx-whisper +faster-whisper +useful-moonshine-onnx +requests