The Provider Protocol: Bring Your Own Speech Engine

Vox handles runtime concerns such as audio capture, playback handoff, sessions, warm-up scheduling, multi-client coordination, and telemetry. Speech engines stay behind a small provider protocol.

Some teams need a different ASR model. Others want system speech, a local MLX voice, or a remote TTS provider. The right abstraction is not one blessed model. It is a stable runtime around whichever model fits the product.

What a provider is

A provider is an executable that reads newline-delimited JSON-RPC from stdin and writes responses to stdout. Vox spawns it on first use and keeps it alive for the daemon lifetime.

An ASR provider receives an audio-file path and returns text, word timing, and metrics. A TTS provider receives text and returns encoded audio plus metrics. Providers do not own WebSocket connections, app sessions, microphone permissions, or playback.

The protocol surface

All providers can expose models, install, and preload. ASR providers add transcribe. TTS providers add voices and synthesize.

Every transcription or synthesis result includes stage metrics. totalMs is required for both paths, and ASR also requires inferenceMs. Those measurements feed the same Vox telemetry tagged by model, route, client, and voice.

Registration

Register providers in ~/.vox/providers.json:

{
  "providers": [
    {
      "id": "whisper-cpp",
      "kind": "asr",
      "command": ["/usr/local/bin/vox-whisper", "--threads", "4"],
      "models": ["whisper-large-v3"]
    },
    {
      "id": "my-tts",
      "kind": "tts",
      "command": ["bun", "run", "/path/to/provider.ts"],
      "models": ["my-tts:v1"]
    }
  ]
}

Register ASR and TTS as separate entries, even when one executable serves both. Vox routes by model ID and provider kind. The models field is optional when a provider reports models dynamically.

Response shape

The important part of a minimal ASR response looks like this:

respond(req.id, {
  modelId: req.params.modelId,
  text: "transcribed text goes here",
  elapsedMs: elapsed,
  metrics: {
    inferenceMs: elapsed,
    totalMs: elapsed,
  },
});

Keep stdout reserved for protocol messages. Logs belong on stderr.

Telemetry across providers

The metrics object gives Vox the provider-side stage breakdown. Vox records the complete request as a tagged sample in performance.jsonl. The format and dashboard stay consistent across built-in and external providers.

The inferenceMs or synthesisMs value comes from the provider, while totalMs captures the whole provider request. Keep both honest so cold starts and steady-state inference remain distinguishable.

Start from the canonical contract

The current protocol covers both speech directions, dynamic model discovery, preload progress, voice discovery, and explicit metrics. The canonical contract and a runnable template live in the Provider Protocol documentation and examples/provider-template/.