diff --git a/BENCHMARKS.md b/BENCHMARKS.md index e3055d9..0e8b1ee 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -23,6 +23,20 @@ means are skewed by first-request kernel compilation. | mlx-whisper large-v3 | MLX | 1170 ms | 1251 ms | 48/48 | 21/21 | `"thank you"` | | faster-whisper distil-large-v3 | CPU int8 | 4269 ms | 4304 ms | 46/48 | 21/21 | `"thank you"` | +### How to read this + +- **Median** — the typical time to transcribe one command. This is the number + you feel when talking to your assistant. +- **p90** — nine times out of ten it was at least this fast. A p90 close to the + median means consistent; a long tail is worse than the median suggests. +- **Exact** — how many of the 48 clips came back word-for-word correct. +- **Digits** — of the 21 clips containing a number, how many wrote it as `21` + rather than `twenty one`. This matters more than it looks; see + [Model choice](README.md#model-choice-v2-not-v3). +- **Silence** — what came back for three seconds of pure silence. Anything + other than nothing is the model inventing words, which your assistant then + tries to act on. + **Exact** is a strict string match after normalising case, punctuation and whitespace. **Digits** counts how many of the 21 number-bearing clips came back with digits rather than spelled-out words — see @@ -59,6 +73,11 @@ What the numbers say: - **Latency is raw inference**, excluding Wyoming protocol overhead. End to end through this server, expect roughly 15–35 ms on top. - One machine, one run each. Treat differences of a few percent as noise. +- **These are a point-in-time snapshot.** Nothing re-runs them automatically, + and the test suite cannot detect drift, so treat them as stale after any + change to the audio path, a `parakeet-mlx`/MLX upgrade, or a macOS release + (which changes the TTS voices the clips are built from). Re-run the sweep + and update the table in the same commit as the change. ## Reproducing diff --git a/README.md b/README.md index 72b1400..30156ea 100644 --- a/README.md +++ b/README.md @@ -40,41 +40,147 @@ latency). ## Requirements -- Apple Silicon Mac (MLX is Metal/ANE-backed) -- Python 3.10+ **with the `lzma` module** — `librosa` pulls in `pooch`, which - imports it. Pythons built without `xz` (a common pyenv default) pass every - version check and then fail at import time with - `ModuleNotFoundError: _lzma`. Homebrew's Python is fine. -- ~2.3 GB disk for the model, ~600 MB for MLX wheels +- **An Apple Silicon Mac** (M1 or newer) that stays on. This does not run on + Intel Macs, a Raspberry Pi, or inside Home Assistant OS. +- **Homebrew Python.** If you are not sure you have it: + ```bash + brew install python@3.13 + ``` +- **About 3 GB of free disk** — 2.3 GB of model, 600 MB of libraries. -`ffmpeg` is **not** required — Wyoming already delivers 16 kHz mono PCM, so -the mel spectrogram is built directly. +
+Why Homebrew Python specifically + +The installer needs Python 3.10 or newer that includes the `lzma` module. +macOS ships Python 3.9, which is too old, and `pyenv` frequently builds +Python without `lzma` (it needs `xz` present at build time). Both fail — the +pyenv case unhelpfully, passing every version check and then dying at import +time with `ModuleNotFoundError: _lzma`. Homebrew's builds are fine. + +`ffmpeg` is **not** required, unlike most speech-to-text tooling. Wyoming +already delivers audio in the right format, so this builds the mel +spectrogram directly. +
## Install +**1. Download and run the installer.** + ```bash git clone https://github.com/adamhf/wyoming-parakeet-mlx cd wyoming-parakeet-mlx ./install.sh ``` -This creates a virtualenv, runs the unit tests, pre-downloads the model, and -registers a LaunchDaemon on port 7892 that starts at boot without needing a -GUI login. It installs *in place*, so keep the checkout somewhere permanent. +It will ask for your password once, to register the background service. -Options: `--port`, `--model`, `--user`, `--python`, `--no-daemon`, -`--no-download`. +Expect it to take a few minutes — most of that is downloading the model. It +sets up its own Python environment, runs the test suite, downloads the model, +and registers a background service on port 7892 that starts automatically when +the Mac boots, with no need to log in. -Then in Home Assistant: **Settings → Devices & Services → Add Integration → -Wyoming Protocol**, enter the host and port, and select the new engine as the -speech-to-text step of your Assist pipeline. +**Keep this folder where it is.** The installer sets things up inside it, so +moving or deleting the folder afterwards breaks the service. -Remove it with `./uninstall.sh`. +Remove it later with `./uninstall.sh`. -### Server options +
+Installer options -`install.sh` bakes these into the plist; run `script/run --help` to see them -all. The ones worth knowing: +`--port`, `--model`, `--user`, `--python`, `--no-daemon`, `--no-download`. +Run `./install.sh --help` for details. See [Security](#security) for why +`--user` is worth using. +
+ +## Connect it to Home Assistant + +Installing it is not enough on its own — Home Assistant has to be told about +it, and then told to *use* it. That is two separate steps, and the second one +is easy to miss. + +**1. Find your Mac's IP address.** System Settings → Network → click your +active connection → Details → the "IP address" field. Or in a terminal: + +```bash +ifconfig | grep "inet " | grep -v 127.0.0.1 +``` + +That command often lists more addresses than you expect. Ignore anything +belonging to virtualisation software (Parallels, VMware, Docker — commonly +`10.211.55.x`, `10.37.129.x`, `172.17.x.x`) or a VPN (Tailscale uses +`100.x.x.x`); those are not reachable from Home Assistant in the usual case. +You want the one on the same network as Home Assistant itself — if Home +Assistant is on `192.168.1.50`, you want the Mac's `192.168.1.x` address. + +If your Mac has both Wi-Fi and Ethernet active you will see two candidates. +Either works, but pick one that will stay put — a wired address, ideally with +a DHCP reservation on your router — because if that interface goes away, Home +Assistant loses speech-to-text. + +**2. Add the integration.** In Home Assistant: **Settings → Devices & +Services → Add Integration → Wyoming Protocol**. Enter the IP address from +step 1 and port **7892**. + +If it succeeds you get a new entry called `parakeet-mlx`. If it says it cannot +connect, see [Troubleshooting](#troubleshooting). + +**3. Actually use it.** Go to **Settings → Voice assistants**, click your +assistant, and set **Speech-to-text** to `parakeet-mlx`. Save. + +Until you do this, nothing changes — you will have installed it and Home +Assistant will carry on using whatever it used before. If you have more than +one assistant configured, change each one you care about. + +**4. Try it.** Click the Assist icon (top right of Home Assistant), press the +microphone, and say something like "turn off the kitchen lights". + +## Check it's working + +Is the service running? + +```bash +sudo launchctl print system/local.wyoming-parakeet | grep "state =" +``` + +Expect `state = running`. + +Watch it work — leave this running, then speak to Assist: + +```bash +tail -f /tmp/local.wyoming-parakeet.stderr +``` + +Each command produces a line like +`2.70s audio -> 124ms, 45 chars`. The `124ms` is how long transcription took. +The text itself is deliberately not logged; see [Security](#security). + +Test it without speaking, using generated audio: + +```bash +./test/make-clips.sh +.venv/bin/python test/wy-test.py test/clips/*.wav +``` + +Expect ten commands transcribed correctly, `silence.wav` coming back empty, +and times around 100–150 ms. + +## Troubleshooting + +| What you see | What it usually means | +|---|---| +| Home Assistant says **"Failed to connect"** when adding the integration | Wrong IP, or the service is not running. Check `state = running` above, confirm the IP with `ifconfig`, and make sure both machines are on the same network. | +| Integration added fine, but **Assist is no worse and no faster** | You missed step 3 — the assistant is still using its old speech-to-text engine. Settings → Voice assistants → Speech-to-text. | +| **`No matching distribution found for parakeet-mlx`** during install | Wrong Python. Run `brew install python@3.13` and try again, or point the installer at it: `./install.sh --python /opt/homebrew/bin/python3.13`. | +| **`error: service did not start`** at the end of install | Look at `/tmp/local.wyoming-parakeet.stderr` for the reason. If you changed `--user`, re-run `./install.sh` rather than editing the service file by hand. | +| The **first command after a reboot is slow** (~300 ms), then it is fast | Expected. The graphics framework compiles its kernels on first use. It settles by the second or third command. | +| Everything works, then **stops after a few weeks** | Check the Mac has not gone to sleep. System Settings → Displays → Advanced → prevent automatic sleeping. | +| You want to see **what it actually heard** | Add `--debug`, but read [Security](#security) first — the log then contains everything spoken near your microphones. | + +## Server options + +Most people never need these. `install.sh` bakes them into the service +definition, so to change one, re-run the installer rather than editing the +service file by hand. Run `script/run --help` for the full list. | Flag | Default | | |---|---|---|