Names are what a voice agent mishears most

We have written that numbers are the hardest thing a voice agent hears in Norwegian. That holds – the Norwegian way of saying numbers is a local problem no international benchmark picks up. But on the measurements that exist of real agent conversations, numbers are not the worst-performing category. Names are.

Where the errors actually sit

AssemblyAI publishes error rates broken out by entity type, measured against Pipecat's open benchmark. That benchmark is built from real conversations between callers and voice agents rather than read speech. Four models, same audio:

Model	Word error rate	Names	Phone numbers
Universal-3.5 Pro Realtime	6.99%	16.92%	3.55%
Google Chirp3	9.04%	22.10%	4.95%
ElevenLabs Scribe v2	9.76%	38.03%	4.78%
Deepgram Flux	15.58%	39.21%	10.41%

Three things are worth stopping on. Names are the weakest category in all four rows. Names run between roughly four and eight times worse than phone numbers – same model, same audio. And the headline figure does not predict it: Chirp3 and Scribe v2 sit less than a percentage point apart on word error rate, and sixteen percentage points apart on names.

The caveat belongs here as well. These are a vendor's own published numbers, and the audio is English-language conversations. The levels do not transfer to a Nordic phone line, and a leaderboard position is never measured on your own line. The ordering and the mechanism do.

An independent measurement points the same way. In a paper presented at EDM 2026, Trinh, He and Whitehill measure Whisper large-v3 on lecture audio and get a 32.3% word error rate on named entities against 7.0% on everything else in the same recordings. That is a factor of 4.6 – on exactly the words that carry the meaning. The same authors report Whisper large-v3 at 28.9% for person names on ConEC, a benchmark built from earnings calls.

Why a name is harder than a number

A number has a check built in. A name has none. Ten possible values per position, a known length, and often a check digit that gives away that something is wrong. A surname has a vocabulary of hundreds of thousands and no redundancy at all. If the model hears "Bakke" where the caller said "pakke", nothing in the string says it is wrong.

The letters that distinguish names are the ones the line throws away. B and D, P and T, V and Z, F and S differ in frequency bands that ordinary telephony strips out on the way in. Those are the classic confusable pairs in English, and they are the same pairs that turn a Nordic surname into an ordinary word: Bakke into pakke, Dahl into tall, Vold into fold.

Nordic names add a layer English does not have. Æ, Ø and Å in Norwegian and Danish, Ä and Ö in Swedish, are spelling choices rather than sounds a model can hear as separate letters. Sæther, Sether and Seter can sound alike on a poor line. Bjørn and Bjorn are the same sound. Løvås, Lovaas and Lovås are three ways of writing the same family. A model trained mostly on English has no strong reason to put those characters where they belong.

The same name has several lawful spellings. Olsen and Olsson. Andersen and Anderssen. Christiansen and Kristiansen. Even a phonetically perfect transcription does not point unambiguously at one row in your customer records.

And a name is the one thing a caller cannot say a different way. "Could you repeat that?" works for a question. For "Kjærstad" it hands the agent the same signal again, slightly louder.

What works is context – but only filtered context

The main finding in that same paper is worth carrying away, because it is counter-intuitive. The authors had a language model correct the named entities after transcription, and tried three ways of giving it something to correct against:

The whole context document. The name error rate went up, from 32.3% to 43.4%.

A summary of the document. Worse again: 38.6%.

Only the names in the context that sound phonetically similar to what the model wrote, of the same entity type, together with the sentences they appear in. Down to 22.7% – a 30% relative improvement.

More context made it worse. The right context made it better. And on names that did not appear in the context document at all, the method landed at 23.6%, so it did no harm where it could not help.

AssemblyAI reports the same pattern from another angle: passing the agent's own question along to the speech model cut word error rate by 10.2% across 20,000 voice-agent files, and fabrications fell 18.3%. A model that knows the question was "what is your surname?" does not guess at an ordinary word.

Three things to change in the setup

Feed the names in before the caller says them. If the call arrives from a number you recognise, you know the customer's name before she has said a word – and which colleagues she might ask for. Pass that list in as expected terms. But pass a short list, not the whole register: the measurements above say unfiltered context makes things worse.

Pass the question along with the answer. The agent knows what it just asked. If the speech model knows too, it reads the reply as a name rather than as a word.

Never let the agent write down a name it is unsure of. Models return a confidence score per word. Use it as a gate: below the threshold, the agent reads the name back letter by letter, or the call goes to a human. A name written wrongly into a record or an order costs more than thirty extra seconds on the call.

Three questions we get

"Can't we just ask people to spell it?" Spelling helps more in English than in the Nordic languages, because the NATO alphabet – Alfa, Bravo, Charlie – is known widely enough that callers reach for it unprompted. What it does not solve is the special characters. A caller spelling Løvås has to say "O with a slash" or "oe", and one spelling Sjöberg has to explain Ö. Spelling helps most when the agent reads back, least when the caller reads out.

"Would a better model help?" Yes, and more than people expect. The spread in the table above is real: there is far more to gain from picking the right model for names than for phone numbers. The point is that the headline figure does not show where the spread sits. Choose on the entity number, and measure the switch on your own calls.

"Can we skip the name?" Often yes, and it may be the most useful answer here. Order numbers and phone numbers are the categories measured lowest, and they point just as precisely at the row in your records. Ask for the name because it is polite. Look the customer up on the number, because that is what holds.

Threll.ai builds voice agents in Norwegian, Swedish and Danish. No model hears "Kjærstad" correctly every time over a phone line. There are, however, setups where the agent knows it did not hear it – and that difference decides whether the error surfaces in the conversation or in the record.