Context

VoxRange is AI voice infrastructure. One platform carries every voice interaction: AI agents that take and make phone calls, and AgentIntel, which transcribes and scores calls so that every agent conversation gets reviewed, not a sample. I am the lead architect.

The market shaped the product. VoxRange is built for Pakistani telephony, with Urdu and code-switched conversations supported out of the box. A voice product like this needs telephony, transcription and natural speech in real time, and we did not want to depend entirely on per-minute third-party APIs for them.

Constraints

• The audio path is real time. A caller hears every delay as dead air, and every stage adds to it: hearing the caller, deciding what to say, and saying it.
• GPU capacity is self-hosted and finite. Speech-to-text and text-to-speech run on our own GPU hardware, not an elastic cloud API.
• Phone audio is hard to hear. Narrowband phone audio, mixed languages and names make transcription harder than it is on clean recordings.
• Every client is a tenant. Agents, knowledge bases, calls and analysis belong to one organisation and must stay scoped to it.

Architecture

The platform is a TypeScript monorepo of NestJS services and Next.js dashboards, next to a separate Python voice runtime that hosts speech-to-text, the turn-taking orchestrator and text-to-speech.

A call arrives through Asterisk. A NestJS media bridge talks to Asterisk over its REST interface, resamples the audio and streams it to the voice runtime. There, speech-to-text hears the caller, the LLM agent decides what to say, and our own text-to-speech models speak the answer back down the same stream.

Knowledge-base answers come from retrieval over pgvector in PostgreSQL, served by the API. The LLM layer is provider-agnostic: an organisation can use OpenAI, Anthropic or any OpenAI-compatible endpoint, and every LLM call goes through one path.

Everything that can wait leaves the live path. The API hands work to NestJS workers through BullMQ queues on Redis: knowledge-base indexing, transcription and analysis of uploaded calls, and nightly jobs. The client, admin and AgentIntel dashboards are separate Next.js apps over the same API.

Decisions and alternatives I rejected

Embeddings stay on a hosted API

Each knowledge-base turn made a round trip to a hosted embedding API, so I tried to move embeddings into a local service inside the voice runtime. It failed twice for different dependency reasons. While it was half-deployed it broke retrieval, and the agent invented answers instead of using the knowledge base.

I abandoned local embeddings and kept the hosted model for both indexing and live queries. The latency target had already been met by other changes, and the data said the LLM, not the embedding, dominated each turn. A smaller single-file model would have worked, but it lost too much quality.

Emotion is read from the text, not the audio

To make the agent better at sales calls, we built a Smart Agent Builder: a guided interview that produces a structured agent profile, then a compiled prompt that a human reviews before it goes live.

For emotion, I rejected acoustic emotion detection. It was a risky bet for Urdu and would have added another model to the live path. The compiled prompt tells the agent how to respond to frustration, confusion or interest that it reads from the words themselves, which adds no latency.

Separate speakers from the audio, not the text

On uploaded mono recordings, an LLM guessed who said each line, and the labels were often wrong. The design I chose separates voices acoustically: use the channels when a recording is stereo, run speaker diarization when it is mono, keep the LLM split only as a fallback, and let the LLM make one decision about which speaker is the agent instead of labelling every line. The model is baked into the image at build time, a direct lesson from the embeddings attempt.

What went wrong

A filter that hid real speech. Stray Latin-script text kept appearing in Urdu transcripts, so we filtered it out. That also removed real speech: Roman Urdu words and code-switched English. The filter was reverted. Junk transcription has to be fixed at the source, in end-of-speech detection, language anchoring and vocabulary, not cleaned up afterwards.

A tuning value that drifted. The silence the system waits for before deciding a caller has finished speaking is a single environment setting. We had tuned it after earlier call problems, and on a later call it was back at an aggressive value that cut callers off mid-sentence. A value that exists only in a server's environment drifts. The lesson is to keep tuning values in versioned configuration and to measure every stage of a turn, retrieval included, before optimising any of them.

Outcome

VoxRange is production voice AI infrastructure covering telephony, speech-to-text and custom text-to-speech. Real phone calls reach the Urdu agent, which hears the caller, answers from the knowledge base and speaks with our own voice models, running on self-hosted GPUs. AgentIntel reached a feature-complete first version, and clients can bulk-upload their historical calls for analysis through the existing analysis pipeline.