Add benchmark harness and compare against 11 other STT backends

Measures median latency, strict exact-match accuracy, digit formatting and
silence behaviour across 48 clips (16 Home Assistant commands x 3 macOS TTS
voices), on an M4 Mac mini.

Headline: Parakeet v2 at 102ms median is 5x faster than the best whisper.cpp
configuration and lands within one clip of it on accuracy. mlx-whisper
large-v3 is the only backend to score 48/48, at 11x the latency. Moonshine is
2x faster again but gives up real accuracy (35/48).

Also quantifies the reason this defaults to v2 over v3: v3 returned digits
for only 10 of 21 number-bearing commands, against 21/21 for v2, which is
most of the gap between their exact-match scores.

The harness feeds audio to every backend as an array rather than a path --
mlx-whisper and moonshine otherwise shell out to ffmpeg, which this project
deliberately does not require.

Clips are gitignored; bench/make_clips.sh regenerates them. Caveats are
documented in the README: this is clean synthetic TTS, so it measures latency
rigorously and accuracy only as a domain smoke test, and faster-whisper is
CPU-only on Apple Silicon because CTranslate2 has no Metal backend.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitignore

@@ -4,3 +4,5 @@ __pycache__/
 .pytest_cache/
 # Generated by test/make-clips.sh
 test/clips/
+# Generated by bench/make_clips.sh
+bench/clips/

README.md

@@ -10,23 +10,84 @@ bridge and inference.
 
 ## Why
 
-Measured on an M4 Mac mini against ten typical Home Assistant voice commands,
+On an M4 Mac mini, Parakeet transcribes a typical Home Assistant command in
-replacing a whisper.cpp setup:
+**~100 ms** — roughly 5× faster than the best whisper.cpp configuration and
-
+9× faster than whisper `large-v3`, while matching them on accuracy. See
-| Backend | Mean latency | Correct | Silent input |
+[Benchmarks](#benchmarks) for the full comparison against eleven other
-|---|---|---|---|
+backends.
-| whisper.cpp `large-v3` | ~1150 ms | 10/10 | `"Thank you."` |
-| whisper.cpp `large-v3-turbo` | ~570 ms | 10/10 | `"Thank you."` |
-| **parakeet-tdt-0.6b-v2** | **~110 ms** | **10/10** | `""` |
 
 Two things matter beyond raw speed:
 
-- **Silence returns an empty string.** Whisper hallucinates `"Thank you."` on
+- **Silence returns an empty string.** Every whisper variant tested
-  digital silence, which reaches your conversation agent as a real utterance.
+  hallucinates on digital silence (`"Thank you."`, or `"you"` for
+  faster-whisper), which reaches your conversation agent as a real utterance.
+  Parakeet and Moonshine return nothing.
 - **No cross-request contamination.** whisper.cpp's server carries decoder
   context between requests unless you pass `-nc`, and will return the
   *previous* utterance — in testing, roughly one time in five.
 
+## Benchmarks
+
+All figures measured on one machine: **M4 Mac mini (10-core, 32 GB), macOS 26.5**.
+48 clips — 16 Home Assistant commands rendered through three macOS TTS voices
+(Daniel, Samantha, Karen). Every backend loads once, transcribes all 48 clips
+to warm up, then runs a timed pass. Latency is the **median** of that pass;
+means are skewed by first-request kernel compilation.
+
+Reproduce with `./bench/make_clips.sh && ./bench/benchmark.py --backend ...`.
+
+| Backend | Runtime | Median | p90 | Exact | Digits | Silence |
+|---|---|--:|--:|--:|--:|---|
+| moonshine tiny | ONNX CPU | 28 ms | 44 ms | 31/48 | 15/21 | `""` |
+| moonshine base | ONNX CPU | 53 ms | 70 ms | 35/48 | 20/21 | `""` |
+| **parakeet-tdt-0.6b-v2** | **MLX** | **102 ms** | **112 ms** | **46/48** | **21/21** | `""` |
+| parakeet-tdt-0.6b-v3 | MLX | 128 ms | 149 ms | 36/48 | 10/21 | `""` |
+| faster-whisper tiny.en | CPU int8 | 182 ms | 204 ms | 44/48 | 21/21 | `"you"` |
+| faster-whisper base.en | CPU int8 | 326 ms | 347 ms | 44/48 | 19/21 | `"you"` |
+| whisper.cpp large-v3-turbo | Metal + CoreML | 518 ms | 537 ms | 47/48 | 21/21 | `"thank you"` |
+| mlx-whisper large-v3-turbo | MLX | 828 ms | 848 ms | 47/48 | 21/21 | `"thank you"` |
+| whisper.cpp large-v3 | Metal + CoreML | 941 ms | 1042 ms | 47/48 | 21/21 | `"thank you"` |
+| faster-whisper small.en | CPU int8 | 974 ms | 1032 ms | 47/48 | 21/21 | `"you"` |
+| mlx-whisper large-v3 | MLX | 1170 ms | 1251 ms | 48/48 | 21/21 | `"thank you"` |
+| faster-whisper distil-large-v3 | CPU int8 | 4269 ms | 4304 ms | 46/48 | 21/21 | `"thank you"` |
+
+**Exact** is a strict string match after normalising case, punctuation and
+whitespace. **Digits** counts how many of the 21 number-bearing clips came
+back with digits rather than spelled-out words — see
+[Model choice](#model-choice-v2-not-v3) for why that matters more than it looks.
+**Silence** is the output for three seconds of digital silence.
+
+What the numbers say:
+
+- **Parakeet v2 has the best latency/accuracy trade-off here.** It is 5×
+  faster than the best whisper.cpp configuration and lands within one clip of
+  it on accuracy.
+- **`mlx-whisper large-v3` is the accuracy ceiling** — the only backend to
+  score 48/48 — but costs 11× the latency to get there.
+- **Moonshine is genuinely faster**, at 2× Parakeet's speed, and it also
+  handles silence cleanly. It gives up real accuracy for it (35/48), so it is
+  the right pick only if latency dominates everything else.
+- **Parakeet v3's 10/21 on digits** is the ITN problem quantified. Its exact
+  match (36/48) is dragged down almost entirely by that one behaviour.
+- Both clips Parakeet v2 misses are the same word — "aircon", which the TTS
+  voices render as "air con" / "aircan". Accuracy differences at the top of
+  this table are concentrated in a couple of awkward tokens, not spread out.
+
+### Caveats — read these before trusting the table
+
+- **This is not a WER benchmark.** The audio is clean synthetic TTS from three
+  similar English voices, with no noise, accents, crosstalk or far-field
+  effects. It measures latency rigorously and accuracy only as a domain smoke
+  test. For real word error rates see the
+  [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard).
+- **faster-whisper is CPU-only on Apple Silicon.** CTranslate2 has no Metal
+  backend, so those rows show CPU int8 performance. On an NVIDIA GPU they
+  would look completely different — do not read this as a verdict on
+  faster-whisper generally, only on what it does on this hardware.
+- **Latency is raw inference**, excluding Wyoming protocol overhead. End to
+  end through this server, expect roughly 15–35 ms on top.
+- One machine, one run each. Treat differences of a few percent as noise.
+
 ## Requirements
 
 - Apple Silicon Mac (MLX is Metal/ANE-backed)
@@ -73,8 +134,12 @@ multilingual, because of inverse text normalisation:
 Home Assistant's local intent matching (hassil) expects digits. If your
 pipeline has `prefer_local_intents` enabled, a model that spells numbers out
 still *looks* accurate while quietly pushing commands off the fast local path
-onto your LLM fallback. v3 is the better choice if you need languages other
+onto your LLM fallback.
-than English — just be aware of the trade.
+
+Measured on the benchmark corpus, v3 returned digits for only **10 of 21**
+number-bearing commands, against **21/21** for v2. That single behaviour is
+most of the gap between their exact-match scores. v3 remains the right choice
+if you need languages other than English — just size the trade-off first.
 
 ## Updating the model
 

bench/benchmark.py

@@ -0,0 +1,235 @@
+#!/usr/bin/env python3
+"""Benchmark speech-to-text backends on the Home Assistant command corpus.
+
+Each backend is loaded once, warmed on every clip, then timed. Reported
+latency is the median of the timed pass, which is what a voice assistant
+actually experiences -- means are skewed by Metal kernel compilation on the
+first request.
+
+Usage:
+    ./benchmark.py --backend parakeet:mlx-community/parakeet-tdt-0.6b-v2
+    ./benchmark.py --backend mlx-whisper:mlx-community/whisper-large-v3-turbo
+    ./benchmark.py --backend faster-whisper:base.en
+    ./benchmark.py --backend moonshine:moonshine/base
+    ./benchmark.py --backend whispercpp:http://127.0.0.1:8910/inference
+"""
+import argparse
+import json
+import re
+import statistics
+import sys
+import time
+import unicodedata
+from pathlib import Path
+
+BENCH_DIR = Path(__file__).resolve().parent
+CLIPS = BENCH_DIR / "clips"
+
+# Spoken-form numbers a model might emit instead of digits. Home Assistant's
+# local intent matching wants digits, so we score this separately.
+WORD_NUMBERS = re.compile(
+    r"\b(one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve|"
+    r"thirteen|fourteen|fifteen|sixteen|seventeen|eighteen|nineteen|twenty|"
+    r"thirty|forty|fifty|sixty|seventy|eighty|ninety|hundred|percent)\b"
+)
+
+
+def normalise(text: str) -> str:
+    """Fold away differences that do not change the intent: case, smart
+    quotes, punctuation, and whitespace. '%' is kept -- it is semantic."""
+    text = unicodedata.normalize("NFKD", text).lower().strip()
+    text = text.replace("’", "'").replace("‘", "'")
+    text = re.sub(r"[^\w\s%']", " ", text)
+    return re.sub(r"\s+", " ", text).strip()
+
+
+def load_corpus():
+    rows = []
+    for line in (BENCH_DIR / "corpus.tsv").read_text().splitlines():
+        if not line.strip() or line.startswith("#"):
+            continue
+        _tts, expected, has_number = line.split("\t")
+        rows.append((expected, has_number == "1"))
+    return rows
+
+
+# --- backends ---------------------------------------------------------------
+
+
+def read_wav(path):
+    """Read a 16kHz mono 16-bit WAV as float32 in [-1, 1).
+
+    Several of these libraries shell out to ffmpeg to load audio, which is an
+    unnecessary dependency when the clips are already in the right format --
+    and unavailable on the benchmark machine. Feed them arrays instead.
+    """
+    import wave
+
+    import numpy as np
+
+    with wave.open(str(path), "rb") as w:
+        assert w.getframerate() == 16000 and w.getnchannels() == 1
+        pcm = w.readframes(w.getnframes())
+    return np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0
+
+
+def backend_parakeet(model_id):
+    import mlx.core as mx
+    from parakeet_mlx import from_pretrained
+    from parakeet_mlx.audio import get_logmel
+
+    model = from_pretrained(model_id)
+
+    def transcribe(path):
+        mel = get_logmel(mx.array(read_wav(path)), model.preprocessor_config)
+        results = model.generate(mel)
+        return results[0].text if results else ""
+
+    return transcribe
+
+
+def backend_mlx_whisper(model_id):
+    import mlx_whisper
+
+    def transcribe(path):
+        return mlx_whisper.transcribe(
+            read_wav(path), path_or_hf_repo=model_id, language="en", fp16=True
+        )["text"]
+
+    return transcribe
+
+
+def backend_faster_whisper(model_id):
+    from faster_whisper import WhisperModel
+
+    # Metal is unsupported by CTranslate2; int8 on CPU is the fastest option
+    # available on Apple Silicon and is what the HA add-on uses by default.
+    model = WhisperModel(model_id, device="cpu", compute_type="int8")
+
+    def transcribe(path):
+        segments, _info = model.transcribe(str(path), language="en", beam_size=5)
+        return "".join(s.text for s in segments)
+
+    return transcribe
+
+
+def backend_moonshine(model_id):
+    import moonshine_onnx
+
+    model = moonshine_onnx.MoonshineOnnxModel(model_name=model_id)
+    tokenizer = moonshine_onnx.load_tokenizer()
+
+    def transcribe(path):
+        # Bypass moonshine_onnx.transcribe() so we can supply the audio as an
+        # array; it expects shape [batch, samples].
+        audio = read_wav(path).reshape(1, -1)
+        return " ".join(tokenizer.decode_batch(model.generate(audio)))
+
+    return transcribe
+
+
+def backend_whispercpp(url):
+    import requests
+
+    def transcribe(path):
+        with open(path, "rb") as fh:
+            r = requests.post(
+                url,
+                files={"file": fh},
+                data={"response_format": "json", "no_context": "true"},
+                timeout=120,
+            )
+        r.raise_for_status()
+        return r.json()["text"]
+
+    return transcribe
+
+
+BACKENDS = {
+    "parakeet": backend_parakeet,
+    "mlx-whisper": backend_mlx_whisper,
+    "faster-whisper": backend_faster_whisper,
+    "moonshine": backend_moonshine,
+    "whispercpp": backend_whispercpp,
+}
+
+
+# --- runner -----------------------------------------------------------------
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--backend", required=True, help="kind:model_or_url")
+    ap.add_argument("--label", help="Name to report (defaults to --backend)")
+    ap.add_argument("--json", action="store_true", help="Emit a JSON result line")
+    args = ap.parse_args()
+
+    kind, _, target = args.backend.partition(":")
+    if kind not in BACKENDS:
+        sys.exit(f"unknown backend {kind!r}; pick one of {', '.join(BACKENDS)}")
+
+    corpus = load_corpus()
+    clips = sorted(
+        (p for p in CLIPS.glob("*.wav") if p.name != "silence.wav"),
+        key=lambda p: (int(p.stem.split("_")[0]), p.stem),
+    )
+    if not clips:
+        sys.exit("no clips found -- run ./make_clips.sh first")
+
+    load_started = time.monotonic()
+    transcribe = BACKENDS[kind](target)
+    load_seconds = time.monotonic() - load_started
+
+    # Warm every clip first: the first inference compiles kernels, and clip
+    # length varies enough that a single warm-up does not cover all shapes.
+    for clip in clips:
+        transcribe(clip)
+
+    latencies, exact, number_ok, number_total, failures = [], 0, 0, 0, []
+    for clip in clips:
+        index = int(clip.stem.split("_")[0]) - 1
+        expected, has_number = corpus[index]
+
+        started = time.monotonic()
+        got = transcribe(clip)
+        latencies.append((time.monotonic() - started) * 1000)
+
+        if normalise(got) == normalise(expected):
+            exact += 1
+        else:
+            failures.append((clip.name, normalise(expected), normalise(got)))
+        if has_number:
+            number_total += 1
+            if not WORD_NUMBERS.search(normalise(got)):
+                number_ok += 1
+
+    silence = CLIPS / "silence.wav"
+    silence_out = normalise(transcribe(silence)) if silence.exists() else "n/a"
+
+    label = args.label or args.backend
+    result = {
+        "backend": label,
+        "clips": len(clips),
+        "median_ms": round(statistics.median(latencies), 1),
+        "p90_ms": round(sorted(latencies)[int(len(latencies) * 0.9)], 1),
+        "exact_match": f"{exact}/{len(clips)}",
+        "digits_ok": f"{number_ok}/{number_total}",
+        "silence": silence_out,
+        "load_s": round(load_seconds, 1),
+    }
+
+    if args.json:
+        print(json.dumps(result))
+    else:
+        print(f"\n=== {label} ===")
+        for key, value in result.items():
+            if key != "backend":
+                print(f"  {key:12s} {value}")
+        if failures:
+            print(f"  mismatches ({len(failures)}):")
+            for name, want, got in failures[:8]:
+                print(f"    {name}\n      want: {want}\n      got:  {got}")
+
+
+if __name__ == "__main__":
+    main()

bench/corpus.tsv

@@ -0,0 +1,17 @@
+# tts_input	expected_transcript	has_number
+turn off the kitchen lights	turn off the kitchen lights	0
+set the living room thermostat to twenty one degrees	set the living room thermostat to 21 degrees	1
+what is the temperature in the bedroom	what is the temperature in the bedroom	0
+dim the hallway lights to thirty percent	dim the hallway lights to 30%	1
+is the back door locked	is the back door locked	0
+turn on the christmas tree in the conservatory	turn on the christmas tree in the conservatory	0
+set a timer for twelve minutes	set a timer for 12 minutes	1
+play radio six music in the kitchen	play radio 6 music in the kitchen	1
+what's the octopus agile rate right now	what's the octopus agile rate right now	0
+close the blinds in the study and turn on the desk lamp	close the blinds in the study and turn on the desk lamp	0
+turn the bedroom lights down to five percent	turn the bedroom lights down to 5%	1
+how much solar am I generating	how much solar am i generating	0
+lock the front door and turn off all the lights	lock the front door and turn off all the lights	0
+set the upstairs aircon to eighteen degrees	set the upstairs aircon to 18 degrees	1
+pause the music in the living room	pause the music in the living room	0
+remind me in forty five minutes	remind me in 45 minutes	1

bench/make_clips.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# Render the benchmark corpus to 16kHz mono WAV using macOS TTS, across
+# several voices. Synthetic speech is clean and accent-consistent, so treat
+# the accuracy numbers as a domain smoke test, not a WER benchmark.
+set -euo pipefail
+
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+OUT="$DIR/clips"
+VOICES=(Daniel Samantha Karen)
+
+rm -rf "$OUT"; mkdir -p "$OUT"
+
+i=0
+while IFS=$'\t' read -r phrase _expected _has_number; do
+    [[ "$phrase" =~ ^#  || -z "$phrase" ]] && continue
+    i=$((i + 1))
+    for voice in "${VOICES[@]}"; do
+        if ! say -v "$voice" -o "$OUT/${i}_${voice}.aiff" "$phrase" 2>/dev/null; then
+            say -o "$OUT/${i}_${voice}.aiff" "$phrase"
+        fi
+        afconvert -f WAVE -d LEI16@16000 -c 1 \
+            "$OUT/${i}_${voice}.aiff" "$OUT/${i}_${voice}.wav"
+        rm -f "$OUT/${i}_${voice}.aiff"
+    done
+done < "$DIR/corpus.tsv"
+
+python3 - "$OUT/silence.wav" <<'PY'
+import sys, wave
+w = wave.open(sys.argv[1], "wb")
+w.setparams((1, 2, 16000, 0, "NONE", "NONE"))
+w.writeframes(b"\x00\x00" * 16000 * 3)
+w.close()
+PY
+
+echo "Wrote $(ls "$OUT"/*.wav | wc -l | tr -d ' ') clips to $OUT"

bench/requirements.txt

@@ -0,0 +1,7 @@
+# Backends compared in the README benchmark table. Install into a throwaway
+# venv -- not the service venv.
+parakeet-mlx
+mlx-whisper
+faster-whisper
+useful-moonshine-onnx
+requests