🎙️OpenAI's Voice Agents Are Fighting WebRTC, and WebRTC Keeps Dropping Packets
Your voice agent is fighting a protocol built for Zoom
TL;DR
WebRTC was built for Google's conferencing product, and it drops late audio packets by default to keep calls real-time. Voice AI agents don't need that tradeoff, since they can process and generate audio faster than real time. If you're building voice agents, the transport layer is now a real bottleneck worth rethinking.
OpenAI still has a WebRTC problem, and it's not a small one. WebRTC was designed for human conference calls, which means it drops late audio packets to keep latency feeling real-time. That's the right call for two people talking on a video call. It's the wrong call for a voice agent. Voice agents can hear and speak faster than real time. Speech-to-text can process 20x speed without trouble, and TTS can generate audio far faster than playback. But WebRTC sends at human speed and gives up on anything late, so the pipeline is artificially throttled by a protocol that assumes a human is on the other end. If you're shipping voice features on top of LLMs, your latency budget is getting eaten by transport decisions you probably never looked at. The numbers make the mismatch clear. One second of Opus at 64Kbps is about 8KB, and two minutes of audio is roughly 1MB, so bandwidth is not the constraint. WebRTC also renders audio based on arrival time rather than timestamps, while most other protocols lean on timestamps. The fix people are circling is MoQ-style delivery: send everything ASAP and let the player buffer, since RAM is cheap enough to hold audio locally.

Key Points
WebRTC drops any late audio packets by default to keep latency real-time, a tradeoff designed for human callers rather than AI agents that can process far faster than speech.
Keep reading — it's free
Enter your email to keep reading — plus the best of AI & tech, daily. Free, forever.
Already a member? Sign in
Comments
Be the first to comment