The Mic Opens in 0.44 Seconds

By Ali AlQattan. Talking to my coding agents felt like a walkie-talkie. Most of the delay turned out to be the audio player.

TL;DR

Vox is the voice layer I built for my coding agents. It runs entirely on my Mac: whisper.cpp hears me, Kokoro speaks, and a daemon decides whose turn it is. The gap between the agent's last word and the microphone opening used to sit around 1.2 seconds. Now it is 0.44 s. Most of the old gap was the program playing the audio, plus a third of a second of silence at the end of every clip.

What Vox is

Vox is a local-only voice runtime with an MCP server in front of it, so Claude Code, Codex, or any MCP client can talk and listen through one tool call. The audio engine lives in a persistent daemon. Each MCP server is a small adapter that can disconnect and reconnect without tearing the engine down, so restarting an agent does not kill voice.

A few rules shape everything else:

• Speech stays local. The Whisper and Kokoro endpoints must be localhost or a literal loopback address, and Vox refuses to start if they aren't. No cloud speech calls.
• One explicit lifecycle. OFF → IDLE → LISTENING or SPEAKING → IDLE. No hidden state machine inside a subprocess.
• Turns queue. If two agents want to talk, the second waits in FIFO order instead of failing. Waiting is capped at 30 s, and cancel or stop drains the queue.
• The mic is honest. One capture stream per turn with a software gate in front of it. When the gate is shut, frames are dropped in the realtime callback. Nothing is queued, buffered or classified.

There are global hotkeys too. Holding ⌘§ dictates at the cursor in any app. Tapping it while an agent is talking takes the floor; otherwise it reads the selection aloud. A lock screen pauses voice and unlocking resumes it, so no agent opens the mic on a Mac I walked away from.

That is the plumbing. This article is about the part that decides whether it feels like a conversation.

Where the 1.2 seconds went

In a spoken turn there is one moment that matters more than any other: the agent finishes its sentence and I start mine. If the mic isn't open yet, I either talk into nothing or I wait and learn to wait. Either way the rhythm is gone.

I had already measured the turn as a whole. Vox owned 2 to 4 seconds of it. The agent owned 25 to 40, reading files and composing tables in the middle of what was supposed to be a chat. That part belonged in the instructions the agent reads, not in code. The hand-off was mine.

The log showed a stubborn gap between the last spoken word and the gate opening for my reply. It hovered around 1.2 s.

So I timed afplay, the macOS command-line player Vox used for every clip. A 0.5 s clip, a 2 s clip and a 5 s clip all took their own length plus 0.85 to 0.93 seconds of wall clock. A fixed cost of about 0.85 s per clip, no matter how long it was.

That cost was paid twice. Replies are streamed sentence by sentence (the next sentence synthesizes while the current one plays), so every sentence boundary paid it. And the last clip paid it again before the listen could even start.

I had seen this before and not recognised it. A month earlier I found that afplay took 1.09 s of wall clock to play the 0.16 s "talk now" chime. On 855 turns after a spoken reply, the log put 1.18 s between "listening started" and "gate opened". Playing the chime in process brought that to 0.27 s. I fixed the chime and left the voice on the same slow path.

Playing audio in process

The fix was to stop launching a player at all. Vox now plays clips through PortAudio inside its own process. A clip ends when its last buffer drains, so the hand-off follows the voice instead of a subprocess exit.

Measured live after the change: playback overhead dropped to 0.06 s.

Two safety nets stayed. If a clip isn't PCM (the say fallback), or the output device won't open, Vox falls back to afplay. Setting VOX_INPROCESS_PLAYBACK=0 turns the in-process player off.

The silence after the last word

Kokoro ends every clip with about 0.36 s of digital quiet. Between sentences that is good. It is the pause a listener expects before the next thought. After the final sentence it is just dead air I sit through before the mic opens.

So only the last span of a reply gets its tail trimmed, down to 80 ms. Everything before it keeps its pause.

Put together, the in-process player and the trimmed tail took the last-word-to-mic-open time from about 1.2 s to 0.44 s. That shipped with 515 tests passing.

Then it clicked

The same day, a new problem showed up: a faint click between words.

The PortAudio callback is Python. It shares the interpreter lock with capture, the event loop and the level meter that drives the waveform on screen. The low-latency default left about 10 ms of slack, and every stall longer than that came out as a click.

A bigger buffer (2048 frames, high latency) removed it. What I said, listening live: "there is no noise anymore". The extra latency only delays when a clip starts, not the hand-off after it ends, so the 0.44 s held. Underflows are now counted and logged, so if it comes back the next report is a number and not a feeling.

Opening fast is only half of it

A mic that opens quickly is useless if it closes on you. Most of the work on Vox went into the other side of the turn: deciding when I am done talking.

A fixed silence window is wrong for everything. It started at 1.2 s, which cut me off mid-thought. I raised it to 1.6 s, which made a one-word answer wait as long as a paragraph. Now it scales with how much I've actually said, from 0.6 s up to 1.6 s over 1.5 to 3 seconds of measured speech. It keys on speech time, not wall time, so pausing to think in a long answer doesn't drop me onto the short deadline.

A breath is not a sentence. The scaling had a bad edge. The less you said, the faster it hung up, which gave the shortest deadline to the most likely false start. The log had two turns in one minute that ended after 0.10 s and 0.36 s of speech. Under half a second of speech, the full window now applies.

The room sets the gate, not a guess. Long turns once refused to end at all. The noise floor started as a guess made before the mic opened. Answering immediately locked it at −60 dBFS in a room measured at −47. Thirteen decibels too low meant my own room tone counted as speech. I replayed my recorded audio through the real endpointer and the turn would have closed 105 to 118 seconds after I stopped. The floor now moves freely until the first real reading lands.

The reply window waits for a voice. After the agent speaks, the mic stays open for 5 seconds of waiting for someone to talk. A click or a chair used to count as "someone", and a 5 s window closed in 1.68 s with 0.08 s of speech in it. A close with less than 0.2 s of speech now throws the audio away and hands the window back.

No guard after the agent's own voice. A cold listen waits out a 1-second open guard (opening the mic stream makes a pop, especially on Bluetooth, that the endpointer would mistake for speech) before it plays the rising blip that means "I can hear you now". Right after Vox has spoken, the stream is already open, so the guard is skipped. The blip still plays, but in process, so the gate opens 0.27 s after the listen starts instead of 1.18 s.

What a conversation actually needs

A conversation is mostly hand-offs. Each one has to be fast, and each one has to be right. Fast without right means the mic opens and then cuts you off. Right without fast means you learn to wait, and once you're waiting you're using a walkie-talkie.

Measure before guessing. Every fix here started from a number: the event log, or my own recorded audio replayed through the real endpointer. The 0.85 s was there the whole time, in a tool I had already caught being slow once.

And let the person win ties. Tap a key and you have the floor. With headphones on and barge-in enabled, talk over the agent and it stops mid-word, with your opening words kept in a 300 ms pre-roll so you don't have to repeat them. Lock the screen and nobody is listening.

0.44 s isn't instant. It is under half of what it was, and the audio player is only 0.06 s of it.