Fix speech at the source

VoxRange runs AI agents that answer real phone calls in Urdu and in the code-switched mix of Urdu and English that people actually speak. I am the lead architect. This post is about one mistake we made in the hearing part of that pipeline, and the rule I took from it.

The symptom

Speech-to-text on Urdu calls kept producing stray Latin-script text in the middle of Urdu transcripts. Narrowband phone audio, mixed languages and names already make transcription hard, and these fragments made it worse: whatever the transcript says is what the agent responds to.

The obvious move was to clean the output. If an agent is set up for Urdu and a fragment comes back in Latin script, drop it before the agent sees it.

Why the filter was wrong

The filter worked, in the sense that the junk disappeared. So did real speech.

Callers do not stay inside one script. Plenty of Urdu is written in Latin letters as Roman Urdu, and plenty of everyday conversation switches into English for a word or a whole phrase. An invented example: "mera appointment confirm kar dein" is an ordinary request, and every word of it is real speech in Latin letters. A rule that removes Latin-script text cannot tell that sentence from noise, because by the time it runs the only thing left to judge is the script, and the script is not the problem.

We reverted the filter. A narrower one stayed: scripts that cannot plausibly be real speech on these calls, such as Cyrillic or CJK characters, are still dropped, because removing them costs nothing real.

Where the junk actually comes from

The stray fragments were a symptom of the hearing stage misbehaving, so that is where the fix belongs:

• End-of-speech detection. Voice activity detection decides when a caller has stopped talking. If it decides too early, a turn is cut into fragments, and a fragment of a word is exactly the kind of audio a speech model turns into confident nonsense.
• Language anchoring. Telling the model which language to expect, instead of letting it guess on every short clip, keeps it from wandering into another language on a burst of noise.
• Vocabulary. Names, places and domain words that the model has never seen come back as near-miss guesses. Giving it the vocabulary it will hear is cheaper than repairing the guesses.

Each of these changes what the model hears or expects. None of them throws away anything the caller said.

The related lesson: settings drift

End-of-speech detection depends on one value: how much silence to wait for before deciding the caller has finished. We had tuned it after earlier call problems. On a later call it was back at an aggressive value, and the agent was cutting callers off mid-sentence. The fragmented turns that produced bad transcripts in the first place were back too.

The value lived only in a server's environment. Nothing in the code or its history held the tuned value, so nothing stopped it from changing. A tuning value like this belongs in versioned configuration, next to the code that reads it, with a note about why it is set where it is:

# voice-runtime config, versioned with the code
# How long to wait in silence before ending the caller's turn.
# Too short cuts callers off and fragments transcripts.
END_OF_SPEECH_SILENCE_MS: <tuned value>

The same call taught a second thing. Dead air on a turn is the sum of several stages: waiting for end of speech, transcription, retrieval, the LLM, and speech synthesis. Some of those were timed and one was not. You cannot decide what to optimise until every stage of a turn is measured, because the stage you are not measuring is the one you will end up guessing about.

What I would tell someone building the same thing

1. When a model produces junk, find the stage that caused it before writing a rule to remove it. Output filters cannot see causes.
2. Test any cleanup rule against real, messy speech, including code-switching, before trusting it.
3. Keep tuning values in versioned configuration with a reason attached, not only in a server's environment.
4. Measure every stage of a turn before optimising any of them.