Expand benchmark reproduction instructions
Give the exact command behind every table row, document the whisper.cpp server setup those rows need (including why -nc matters for a fair comparison), and state the venv/model cost plus cleanup. Call out that bare 'python3' is the system 3.9 on macOS and fails with 'No matching distribution found for parakeet-mlx' -- verified. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -34,7 +34,7 @@ All figures measured on one machine: **M4 Mac mini (10-core, 32 GB), macOS 26.5*
|
|||||||
to warm up, then runs a timed pass. Latency is the **median** of that pass;
|
to warm up, then runs a timed pass. Latency is the **median** of that pass;
|
||||||
means are skewed by first-request kernel compilation.
|
means are skewed by first-request kernel compilation.
|
||||||
|
|
||||||
Reproduce with `./bench/make_clips.sh && ./bench/benchmark.py --backend ...`.
|
Every row is reproducible — see [Reproducing](#reproducing) below.
|
||||||
|
|
||||||
| Backend | Runtime | Median | p90 | Exact | Digits | Silence |
|
| Backend | Runtime | Median | p90 | Exact | Digits | Silence |
|
||||||
|---|---|--:|--:|--:|--:|---|
|
|---|---|--:|--:|--:|--:|---|
|
||||||
@@ -88,6 +88,90 @@ What the numbers say:
|
|||||||
end through this server, expect roughly 15–35 ms on top.
|
end through this server, expect roughly 15–35 ms on top.
|
||||||
- One machine, one run each. Treat differences of a few percent as noise.
|
- One machine, one run each. Treat differences of a few percent as noise.
|
||||||
|
|
||||||
|
### Reproducing
|
||||||
|
|
||||||
|
**1. Set up a throwaway venv.** Keep the benchmark dependencies out of the
|
||||||
|
service venv — they pull in CTranslate2, ONNX Runtime and a second copy of
|
||||||
|
MLX, none of which the server needs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3.13 -m venv /tmp/bench-venv # same requirement as install.sh:
|
||||||
|
/tmp/bench-venv/bin/pip install -r bench/requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
Use an explicit interpreter, not bare `python3` — on macOS that is still the
|
||||||
|
system Python 3.9, which has no MLX wheels and fails with
|
||||||
|
`No matching distribution found for parakeet-mlx`. Homebrew's
|
||||||
|
`python3.11`/`3.12`/`3.13`/`3.14` all work.
|
||||||
|
|
||||||
|
**2. Render the clips** (macOS only — it uses the built-in `say` voices):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./bench/make_clips.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Writes 49 files to `bench/clips/` (16 phrases × 3 voices, plus silence). They
|
||||||
|
are gitignored; regenerating them is deterministic for a given macOS release,
|
||||||
|
but voice quality does change between releases, so numbers are only strictly
|
||||||
|
comparable within one machine.
|
||||||
|
|
||||||
|
**3. Run a backend.** The `--backend` argument is `kind:model`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
/tmp/bench-venv/bin/python bench/benchmark.py \
|
||||||
|
--backend parakeet:mlx-community/parakeet-tdt-0.6b-v2
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `--json` for a single machine-readable line instead of the human report,
|
||||||
|
and `--label` to control the name in the output. Without `--json` it also
|
||||||
|
prints every mismatch with expected-vs-actual, which is the useful bit when a
|
||||||
|
model scores worse than you expected.
|
||||||
|
|
||||||
|
If you have `HF_HUB_OFFLINE=1` exported (the service sets it), prefix the
|
||||||
|
command with `HF_HUB_OFFLINE=` so it can fetch models it has not seen.
|
||||||
|
|
||||||
|
**The exact commands behind each table row:**
|
||||||
|
|
||||||
|
| Row | Command |
|
||||||
|
|---|---|
|
||||||
|
| moonshine tiny | `--backend moonshine:moonshine/tiny` |
|
||||||
|
| moonshine base | `--backend moonshine:moonshine/base` |
|
||||||
|
| parakeet-tdt-0.6b-v2 | `--backend parakeet:mlx-community/parakeet-tdt-0.6b-v2` |
|
||||||
|
| parakeet-tdt-0.6b-v3 | `--backend parakeet:mlx-community/parakeet-tdt-0.6b-v3` |
|
||||||
|
| faster-whisper tiny.en | `--backend faster-whisper:tiny.en` |
|
||||||
|
| faster-whisper base.en | `--backend faster-whisper:base.en` |
|
||||||
|
| faster-whisper small.en | `--backend faster-whisper:small.en` |
|
||||||
|
| faster-whisper distil-large-v3 | `--backend faster-whisper:distil-large-v3` |
|
||||||
|
| mlx-whisper large-v3-turbo | `--backend mlx-whisper:mlx-community/whisper-large-v3-turbo` |
|
||||||
|
| mlx-whisper large-v3 | `--backend mlx-whisper:mlx-community/whisper-large-v3-mlx` |
|
||||||
|
| whisper.cpp \* | `--backend whispercpp:http://127.0.0.1:8920/inference` |
|
||||||
|
|
||||||
|
\* The whisper.cpp rows need a server running first. That requires your own
|
||||||
|
[whisper.cpp](https://github.com/ggml-org/whisper.cpp) build — ideally with
|
||||||
|
`-DWHISPER_COREML=1` for the CoreML encoder, which is what the table
|
||||||
|
measured:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./build/bin/whisper-server -m models/ggml-large-v3-turbo.bin \
|
||||||
|
--host 127.0.0.1 --port 8920 -nc -sns -l en -t 4
|
||||||
|
```
|
||||||
|
|
||||||
|
`-nc` matters. Without it the server carries decoder context between requests
|
||||||
|
and periodically returns the previous utterance, which would make whisper.cpp
|
||||||
|
look far worse than it is. The table gives it its best configuration
|
||||||
|
deliberately.
|
||||||
|
|
||||||
|
**Cost.** The full sweep is roughly 15 minutes of compute, dominated by
|
||||||
|
`distil-large-v3` at ~4.3 s per clip, plus however long it takes to pull about
|
||||||
|
7 GB of models. To clean up afterwards:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rm -rf /tmp/bench-venv bench/clips
|
||||||
|
rm -rf ~/.cache/huggingface/hub/models--mlx-community--whisper-large-v3*
|
||||||
|
rm -rf ~/.cache/huggingface/hub/models--Systran--faster-*
|
||||||
|
rm -rf ~/.cache/huggingface/hub/models--UsefulSensors--moonshine*
|
||||||
|
```
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
- Apple Silicon Mac (MLX is Metal/ANE-backed)
|
- Apple Silicon Mac (MLX is Metal/ANE-backed)
|
||||||
|
|||||||
Reference in New Issue
Block a user