Turn taking is the set of decisions about who speaks when: when the caller has finished, when the agent may start, and what happens when both talk at once. It sits underneath the words, so it is invisible in a scripted demo and obvious in the first minute of a real call. Most voice agents that feel wrong are failing here, not at understanding.
What is a turn, exactly?
A turn is one party holding the floor. People hand it back and forth constantly and are extremely good at it, predicting when someone is about to finish from grammar, intonation, breathing and pace, and starting as that finish arrives.
A voice agent approximates the same judgement from audio arriving in one direction, answering three questions continuously:
- Has the caller finished their turn, or just paused?
- Is the sound on the line speech aimed at me, or noise?
- The caller has started talking while I am talking. Should I stop?
None has a clean answer, and all three must be answered without waiting, because waiting is itself an answer and a bad one.
Why is silence detection not enough?
The simplest endpointing rule is to wait for silence and call it the end of the turn. Most systems start there, and it fails in both directions on real calls.
It ends turns too early, because people pause inside sentences: recalling an order number, reading off a screen, deciding how to phrase a complaint. A system that treats every gap as a finished thought talks over them, which is the most irritating thing a voice agent does.
It also ends turns too late, or never, because phone lines are not silent. A car, a kitchen, a shop floor and a speakerphone all deliver a continuous bed of sound. If the rule waits for silence, the silence never comes, and the caller sits there having finished, waiting for a machine that does not know it.
Better endpointing uses more than the absence of sound. It weighs whether what was said sounds grammatically and prosodically complete, whether the pitch fell the way voices fall at the end of a statement, and how the caller has been pacing themselves. It is a prediction, and predictions are sometimes wrong.
What is the difference between a pause and a finished thought?
Consider "My order number is four seven ... hang on ... four seven two one" against "My order number is four seven two one". The middle sounds similar in both. The correct behaviour is completely different.
No setting solves this. What you can do is make the ambiguity rarer by shaping what the agent asks for. One thing at a time gets shorter answers, and short answers have fewer places to pause in. A compound question invites a rambling reply full of mid thought gaps, and the agent then has to guess its way through it.
Why does an agent that cannot be interrupted feel broken?
Barge in is the caller talking over the agent, and the agent stopping to listen. People interrupt constantly on the phone: to correct a wrong assumption, to say they have already done the thing being explained, to answer a question before it finishes. That is normal, cooperative behaviour.
An agent that keeps talking through it fails memorably. The caller now knows they are talking to a machine that does not care what they say, and either shouts or hangs up.
Barge in has its own trap. Handle it too eagerly and the agent stops every time a door closes or a car passes, leaving a trail of half sentences. Telling the caller apart from the caller's environment is genuinely hard on a compressed phone line.
What are backchannels, and why do they break things?
Backchannels are the noises a listener makes to show they are still there: mm hm, right, okay, yeah, achha. They are not attempts to take the floor.
A system that treats every sound from the caller as barge in stops dead every time the caller agrees with it. The conversation becomes a series of aborted sentences, and the caller, who was being polite, has no idea why the agent keeps trailing off.
The reverse matters too. An agent producing none of its own feels absent during a long caller turn, and the caller starts wondering whether the line dropped.
What happens on a noisy line or a speakerphone?
Everything above gets harder, and the reasons stack.
- Background speech. A television or a nearby conversation produces speech shaped audio the agent was never addressed with.
- Speakerphone echo. The agent's own voice returns through the caller's microphone. Mishandled, a system can interrupt itself.
- Codec compression. Phone audio is heavily compressed, and the pitch and breath cues endpointing leans on are partly stripped.
- Packet loss. Network gaps arrive as silence that never happened, and an endpointer reading silence as a finished turn cuts the caller off over a dropped packet.
None of this shows up on a laptop in a quiet office, which is why the first real call always sounds worse than the demo.
What does getting it wrong cost?
It is expensive in both directions, and the two failures need opposite fixes, which is why teams oscillate between them.
| Failure | What the caller experiences | What it costs |
|---|---|---|
| Starts too early | Talked over mid sentence, has to repeat | Caller raises their voice or gives up, and whatever was said during the overlap is lost |
| Starts too late | Dead air after they finish | Caller repeats themselves or says hello, creating a new turn the agent must handle |
| Ignores barge in | Cannot correct or redirect it | Caller concludes it is not listening and asks for a human |
| Treats backchannels as barge in | It keeps trailing off | The conversation never completes a thought |
Note the second row. Dead air is not a neutral failure, because the caller fills it. They say hello or repeat themselves, and now the agent has two overlapping inputs to reconcile. The third row is the one that ends the call: an agent that will not yield gets asked for a human.
Why does a scripted demo hide all of this?
A demo is one person, usually the builder, in a quiet room on a good microphone, speaking in complete sentences, asking the questions the agent was designed for, and politely waiting for it to finish. Every one of those conditions removes a turn taking problem. Your callers do none of it. The gap between a demo and a real call is almost entirely turn taking, which is why a demo is not evidence.
None of this shows up in a demo, so create a free account, build one agent and call it from a moving car. Phone agents describes what the live call path actually does, and since every extra second of hesitation is a second you pay for, pricing is the other reason to care about this.
How do you test turn taking on your own agent?
A few calls will tell you more than a week of transcripts. Make them from a mobile on a normal network.
- Call from a moving car. Windows up is fine. This tests the endpointer against continuous road noise and barge in against passing traffic.
- Interrupt it mid sentence. While the agent is explaining something, start talking. It should stop and act on what you said, not finish its sentence first.
- Pause mid sentence yourself. Say "my booking is for", stop as though thinking, then continue. If the agent jumps into that gap, your questions are inviting turns that are too long.
- Read a long number aloud. An order reference or a policy number, read the way a real person reads it, in groups with gaps. This is where early endpointing hurts most, because a truncated number is worse than no number.
- Say mm hm while it is talking. Check it keeps going instead of stopping dead.
Then read the transcripts and look for cut turns. A truncated caller turn is the fingerprint of an endpointing failure, and it is invisible in the reply, which will be confidently wrong.
What to change in the agent's instructions
Most turn taking behaviour lives below the prompt, but three things you write move it.
- Shorter sentences. Long agent turns give the caller more reason to interrupt and more to lose when they do. Tell the agent to answer briefly and stop.
- One question at a time. The single most effective change available to you. A compound question produces a long, pause filled answer that endpointing will mishandle. Two short questions produce two short answers.
- Confirm long strings in chunks. Have the agent read a reference number back in groups and confirm each group. It gives the caller a natural place to correct a digit, and turns one fragile long turn into several robust short ones.
Keep the wording tight. Instruction fields on a Foan agent share one budget, capped at 4,000 characters combined with a warning at 3,000, so these rules compete with everything else the agent must know.
One thing not to write is a rule about which language to reply in. On the speech to speech path that instruction is unreliable, for a reason related to turn taking: the model commits to a reply before the rule can steer it. That is covered in Hinglish on the phone.
Where to go next
The instruction choices above belong in the first draft rather than a later fix, and how to build an AI phone agent shows where they go. To hear turn taking handled on a live call rather than read about it, start at the voice product page.
Try it on your own number
Build an agent, point a number at it and listen to the first call.