VoxWeft: on-device simultaneous interpretation on Apple Silicon

Simultaneous interpretation usually runs in the cloud: audio up, translation back, a few seconds of round trip, and the conversation leaves the room. VoxWeft is an open-source interpreter that runs on Apple Silicon, Apache-2.0 licensed: speech in, translated speech out, no network. Its core is a port of OpenBMB’s VoxCPM2, a 2B-parameter TTS covering 30 languages (covered earlier), to Apple’s MLX framework, streaming first audio in about 170 ms on an M5 and cloning a voice from about 5 seconds of reference audio.

Core features

• VoxCPM2 on MLX: The README says zero weight conversion for the language model, a bit-exact AudioVAE, and module-by-module verification against the official PyTorch implementation. It supports 4-bit quantization and voice cloning from about 5 seconds of reference audio.
• Cutting while you speak: Per-frame RMS with an adaptive noise floor, a 0.45-second silence rule, a forced cut at the first 100 ms dip after 6 seconds, and a carry buffer that prepends a 0.4-second “So,” to the next segment instead of dropping it. Learned end-of-turn (SmartTurn v3) is wired in but off by default; the author measured it as net-negative for interpretation.
• Any language pair, no pivot: Spanish to Malay or German to Vietnamese goes straight through the LLM in one hop, never via Chinese or English. Only low-resource targets (Lao, Khmer, Burmese) fall to a dedicated MT model.
• Interrupt actually interrupts: A stop cancels the in-flight turn and closes the socket to the TTS, so the next sentence arrives in about 1 second on an M2 instead of queueing 20 seconds behind a dead one.
• Three layers: L1 is the TTS core, usable on its own through voxweft say. L2 adds segmentation, ASR, an LLM and a WebSocket runtime. L3 is a CLI, an MCP server and a roughly 700-line SwiftUI shell.
• An MCP server: Any MCP-speaking agent, Claude Code, Codex or Cursor, can point at the venv inside a clone and gain a local voice.

Benchmarks

These are the README’s own measurements, taken on a base M2 16 GB with background apps killed first:

Metric	M2 16GB	M5 16GB	M3 Ultra
TTS first audio, warm	303 ms	169 ms	124–168 ms
TTS RTF	1.47	0.94	≈ 0.2
Resident memory, all three	6.7 GB	6.7 GB	—
End to end, text to first interpreted audio	—	0.58 / 2.4 / 1.3 s	—

The README is direct about the ceiling: on an M2 the pipeline is TTS-bound, and an RTF of 1.47 means 5 seconds of speech takes 7 seconds to produce. On an M3 Ultra all three models sit under real time and the pipeline streams. It also declines to publish BLEU scores, on the grounds that translation quality belongs to the model you plug in, and what this repository owns is that the pipeline does not lose it on the way.

Typical use cases

• Conversations that must stay private: Medical, legal or internal meetings where the audio never leaves the device.
• Developers giving an agent a local voice: The MCP server connects with one JSON block.
• Researchers in streaming TTS: The MLX port itself is a reusable result, and the README says it was verified module by module against the official implementation.

Quick start

There is no PyPI package, on purpose; the README calls the source the distribution. You need an Apple Silicon Mac and Python 3.10+, and model weights are downloaded separately (docs/MODELS.md has notes, with ModelScope mirrors where Hugging Face is slow):

git clone https://github.com/HenryZ838978/VoxWeft && cd VoxWeft
python -m venv .venv && source .venv/bin/activate
pip install -e .              # TTS only: synthesis, cloning, voxweft say
pip install -e ".[mcp]"       # + MCP server
pip install -e ".[runtime]"   # + segmentation, ASR/LLM adapters, WebSocket server

Summary

VoxWeft suits developers with Apple Silicon who want local interpretation or a local voice, especially those comfortable downloading model weights and reading docs. It does not suit anyone who expects an installer and a finished product: there is no package, the weights are yours to fetch, and the project is days old.

A few things to know first. It is very early: the README’s last commit is dated September 22, 2026, the repository was created on September 20, and it had 10 stars as of October 23, 2026, with no community validation. The default ASR is an 8-bit Qwen3-ASR, and for low-resource languages the README admits the whisper-large-v3-turbo fallback is poor, with Lao sometimes transcribed in Thai script. The optional MT fallback, NLLB-200, is CC-BY-NC-4.0, research use only, and opt-in. Every latency figure is the author’s own measurement under specific conditions, and the README honestly notes that background apps made the same pipeline three to four times slower.

The repository was created on September 20, 2026, is not archived, had 10 stars as of October 23, 2026, and uses the Apache-2.0 license. We checked the README and the repository metadata, and we did not install or run it.