← back to the archiveCover illustration for “A voice transcript needs the moment the instruction changed”
POSTday 57·7w ago·by Andy Padia

A voice transcript needs the moment the instruction changed

When people interrupt an acting voice assistant, the record must preserve the order of speech, action and cancellation. A tidy dialogue is insufficient.

“Wait, don’t submit that” is an instruction whose meaning depends on when it arrived. I would make that sentence part of the acceptance test for any voice assistant allowed to act while it talks.

OpenAI’s GPT-Live introduction describes a full-duplex model that processes incoming audio while producing speech. It can also delegate deeper work to another model while continuing the conversation. Those are useful interaction capabilities. They make the relationship between the conversation and the action worth inspecting separately.

A transcript can preserve overlapping speech if it carries appropriate timing and channel information. Speaker labels are not inherently obsolete. The problem arises when a downstream system turns concurrent events into a neat sequence of alternating paragraphs and treats that sequence as the operational record.

The same words can describe different outcomes

Imagine an assistant preparing a hypothetical travel booking. It says it is submitting the reservation. The user interrupts with the cancellation sentence. Later, a reviewer sees both statements in a transcript.

That text alone may leave several possibilities open. The interruption could have arrived before the booking request was dispatched. It could have arrived while the request was in flight. It could have arrived after confirmation, when stopping submission was no longer possible and cancellation required a different action.

All three conversations can contain nearly identical words. The product owes the user a different response in each case. “Stopped” should mean that the action stopped, not merely that the assistant stopped speaking.

I would preserve the relationship between the input event, the action request and the eventual result. That includes whether an interruption was detected, which pending operation it referred to, and whether cancellation was acknowledged by the system doing the work. A timestamp is useful only when the clocks and event ordering are understood well enough to support the conclusion.

Test interruption against the action

My proposed test uses a harmless mock booking service. Introduce an interruption before dispatch, during a deliberate service delay and after a confirmed result. Inspect the spoken response alongside the service’s event record. The test passes when the explanation matches what actually happened and any unresolved state remains visible.

I would vary the words too. “Hang on” may be a request for conversational attention rather than a clear cancellation. The product needs an explicit policy for ambiguity, particularly when the action has consequences. A smooth acknowledgement should not silently decide that policy for the user.

The launch description does not establish what every downstream export preserves, and I have not audited GPT-Live’s logging or cancellation implementation. This is an integration requirement derived from concurrent interaction, not a claim that OpenAI loses interruptions.

There are practical limits. Some external actions cannot be recalled, some services cannot confirm cancellation immediately, and recording audio may be inappropriate under the applicable policy. The interface must explain those constraints. The event record can still retain permitted action states without pretending that every conversation requires indefinite audio storage.

The archive’s smart-transcription discussion concerns how cleanup changes the words. This issue survives even with a perfectly verbatim transcript. The missing evidence can be the ordering between speech and a tool’s side effect.

That is where I would put the review effort: not simply whether the assistant heard the correction, but whether the correction changed the action in time.

For an acting voice assistant, preserve when the instruction changed and what the system had already done.

#voice-ai#agents#observability#interaction-design
← older drop
Prompt deletion needs a regression record
newer drop →
Buzz swaps the org chart for an audit trail

related drops

explore all 243 drops →
← back to the archiveday 106