GPT-Live splits the latency budget in two

The most useful thing about GPT-Live is not that it can listen and speak at the same time. It is that OpenAI stopped pretending every part of a voice interaction belongs to one latency budget.

Audio has to arrive on a clock. A search, tool call or deeper reasoning step may take seconds. Put both on one path and the slow work eventually damages the live conversation. Split them, and each path can make a promise it can actually keep.

That is the systems lesson I would carry into any enterprise voice agent: define one service level for conversational continuity and another for delegated work. Do not average them into a single “response time” that describes neither experience.

The live path cannot borrow time

OpenAI's systems account describes a live path that moves audio between the client and voice model, plus an asynchronous path for reasoning and tools. The distinction matters because a late audio frame is not merely a slower answer. It can become an audible artifact, an interruption or a broken conversational rhythm.

Startup sits inside that live budget too. OpenAI says a conventional WebRTC setup involved six network round trips. Its WebRTC Abridged Roundtrip Protocol, or WARP, reduces media and data startup to one by overlapping and pre-negotiating parts of the handshake. Instant Connect takes the signalling exchange off the critical path, with the normal flow already running as a fallback.

These are not generic “make it faster” optimisations. They protect the point at which the system must start listening. The design treats a missed deadline as an interaction defect.

The reasoning path needs a different contract

GPT-Live can delegate deeper work to another model while the voice model keeps the conversation moving. That creates a second latency class. The delegated result should arrive quickly, but it does not have to deliver audio frames every few milliseconds.

The mistake would be to treat that freedom as unlimited waiting. A conversational model can acknowledge a request or ask a useful clarification, but filler is not a substitute for progress. The async path needs its own timeout, status policy and terminal action.

For a low-stakes search, the terminal action might be: tell the user the lookup is taking longer and offer to continue later. For an account change, it may be: do not imply success until the tool returns a confirmed state. A smooth voice should never launder an uncertain backend action into apparent completion.

rendering diagram…

Stateful speed changes the failure model

Full-duplex inference keeps working while the user speaks or pauses. Re-reading an entire conversation for every audio frame would be wasteful, so the serving system keeps conversation state close to the model instance processing it. That improves continuity, but it also creates affinity: the session depends on the instance holding its state.

OpenAI describes warming a replacement instance with the conversation before switching traffic. The handoff pattern also supports context compaction: prepare the shortened state elsewhere while the original instance keeps the conversation alive, then switch when the replacement is ready.

This is a good reminder that low latency often comes from retaining state, and retained state makes recovery more demanding. A stateless retry is simple but expensive. A stateful handoff is fast for the user only if the new instance has enough verified context to continue correctly.

I would test that boundary with interruptions, tool calls and handoffs in combination. Start a delegated action, interrupt the voice model, force an instance replacement, then check whether the action status and conversational acknowledgement still agree. Measuring each component in isolation misses the failure that a real user experiences.

Measure two paths and one joined outcome

OpenAI reported a 30 percentage-point improvement over GPT-Realtime-2.1 on Full Duplex Bench. It also reported that Speak saw almost 80% fewer interruptions during learner thinking pauses in early evaluations. Those are useful vendor-reported signals, not a performance guarantee for another product, network or language.

For a deployment, I would keep at least four clocks: connection-to-first-audio, late audio-frame rate, delegation completion time and time until the user receives a confirmed outcome. Then I would segment them by network condition, tool and interruption pattern.

The joined outcome matters because both paths can meet their local targets while the experience still fails. Audio can remain perfectly smooth while a tool result is stale. A tool can complete quickly while the voice model continues speaking over the user's correction. The product succeeds only when the conversational state and action state reconcile.

This advances a different question from my earlier piece on full-duplex transcripts and cancellation. There, the concern was preserving overlapping intent. Here, it is preserving two service promises without letting either hide the other's failure.

What's in it for you

• Give live audio and delegated reasoning separate service levels.
• Put a timeout and explicit fallback on every async action.
• Test handoff, interruption and tool state together, not as isolated components.
• Report the confirmed user outcome alongside component latency.

A realtime voice agent is not one fast pipeline; it is two differently timed systems that must tell the same truth.