The quarter-second that kills a voice agent project
Latency budgets for AI voice, and why your SBC placement matters more than your model choice.
If you want to know if a voice AI deployment is going to fail, you don't need to look at the accuracy of its answers. You just need to listen to the first ten seconds of a conversation.
Human: "Hi, I need to check my account balance."
(Silence for 800 milliseconds)
Human: "Hello? Are you there?"
AI: "I can help with—"
Human: "Oh, sorry, go ahead."
AI: "—checking your balance."
That awkward, stumbling collision is the sound of a broken conversational interface.
In human conversation, the natural gap between one person stopping and the other starting is incredibly brief — roughly 200 to 250 milliseconds. If an AI takes longer than that to respond, it doesn't just sound slow. It sounds unsure. It sounds like a walkie-talkie. Callers will inevitably interrupt it, talk over it, and conclude the system is broken long before they notice how smart the underlying language model is.
That quarter-second latency budget is the most unforgiving metric in voice engineering. Here is where the time actually goes, and why the physical placement of your network edge matters more than the LLM you choose.
The Latency Budget Breakdown
To understand why a voice bot is slow, you have to break the round-trip into its component parts. When a user finishes speaking, a brutal countdown begins:
- Network ingress (20–50ms). The audio packet travels from the user's phone to your enterprise network edge.
- Speech-to-text (50–100ms). The audio is transcribed into text.
- LLM inference, time to first token (100–300ms). The brain of the operation figures out what to say and streams the first word.
- Text-to-speech (50–100ms). That first word is synthesized back into audio.
- Network egress (20–50ms). The audio packet travels back to the user's ear.
If you add those up, you are frequently pushing past 500 milliseconds.
Vendors will often sell you on a "blazing fast LLM" that responds in 50 milliseconds. But if you look at the budget, the LLM is only one piece of the puzzle. You can buy the fastest AI model on earth, but if your network architecture is sloppy, the agent will still sound broken.
Why SBC Placement Dominates the Total
This is where the telecom engineers usually have to intervene and fix what the software developers built.
In a standard enterprise setup, voice traffic from the local telco comes into a Session Border Controller (SBC). If your SBC is hosted in a data center in Singapore, but your AI platform is hosted in a cloud region in the US-West, every single spoken word has to cross the Pacific Ocean, get processed, and cross back.
That geographical hairpin adds a guaranteed 150 to 200 milliseconds of round-trip network latency. You have blown your entire conversational budget before the AI even starts "thinking."
To fix this, SBC placement and media routing are critical. By deploying edge SBCs physically close to the telco break-out, and using media bypass to route the RTP (audio) stream directly to an in-region AI processing hub, we strip the transit time down to bare metal. In voice AI, physics always wins. You cannot out-code the speed of light through a fiber optic cable.
Barge-in and Endpointing: The Illusion of Speed
Beyond the network, perceived speed is heavily dictated by two specific settings: endpointing and barge-in.
Endpointing is the AI's ability to know when you have stopped talking. If the AI waits for two seconds of dead silence to ensure you are finished, the conversation feels agonizingly slow. We spend weeks tuning endpointing algorithms to detect the natural downward inflection of a human sentence, allowing the AI to start processing before the user has technically finished breathing.
Barge-in is the ability for the human to interrupt the AI. For barge-in to work, the system requires aggressive echo cancellation at the SBC or media gateway level. The system has to instantly distinguish between the user speaking, and the sound of its own voice echoing back through the telephone line. If barge-in fails, the AI just keeps talking over the frustrated customer like a relentless telemarketer.
What We Hold Ourselves To
When we architect these solutions, we don't look at average response times. We look at the 95th percentile.
In production, we monitor the RTP streams at the SBC level, measuring the exact millisecond gap between the end of the customer's audio packet and the arrival of the AI's response packet. If that gap consistently drifts above 400 milliseconds, alarms trigger.
The Summary
A conversational AI is only as good as its ability to hold a conversation.
While the industry obsesses over the intelligence of the latest large language models, the true battleground for voice AI adoption is latency. That unforgiving quarter-second budget dictates whether a caller feels heard or frustrated.
By meticulously engineering the media path, anchoring SBCs close to the user, and obsessively tuning endpointing and barge-in settings, we can reclaim the milliseconds lost to the network. Because in the world of voice AI, intelligence doesn't matter if it arrives a second too late.
