The unheard majority: Why voice AI fails most under-represented languages and edge-case customers.

Public Large Language Models (LLMs) are built for the average user and trained in dominant languages like English.

By Alfredo Gemma, AI Solutions Director at Connect

However, when used to train AI voice engines for enterprise customer service needs, these generalist models fail when confronted with real-world edge cases.

The issue is particularly relevant in diverse, multilingual regions, such as South Africa and mainland Europe, as mainstream AI architectures collapse under the weight of sudden language switches, broken speech, background noise, or poor-quality phone lines.

The implications are also commercial because size is costly. A system that must run on specialised graphics hardware in a foreign data centre carries a price tag that makes low-value, high-volume interactions uneconomic to automate, especially when purchasing token-based API services.

As such, improving Voice AI for neglected customer bases requires a fundamental shift away from large, generic models and adopting specialised, full-stack systems designed for real-world complexity.

The misdiagnosed problem.

Ask any voice AI vendor why their system struggles with a Zulu-speaking caller from Soweto in South Africa, a Scots-speaking caller from Glasgow, or a dialect-speaking caller from rural Bavaria, and you will get the same answer: not enough data.

The proposed fix seems reasonable: Wait for the datasets to arrive from the big labs. But this approach is problematic because the reason these systems fail is not a lack of relevant examples. It’s the fact that every component was designed around a different person.

The microphone settings, the pause detector, the way words are broken into pieces, the spell-out rules for numbers, even the scorecard used to declare success were all calibrated on American English recorded in quiet rooms on good equipment..

Data degradation as an asset.

Standard AI training pipelines prioritise pristine, studio-quality audio. However, real-world customer service calls arrive via compressed audio codecs over noisy telephony networks.

To build resilient voice systems, developers must do the exact opposite of audio cleaning: deliberately degrade training data.

By intentionally adding landline and mobile phone noise profiles and applying heavy compression codecs, messy input becomes the most valuable training asset. Models trained on synthetic imperfection easily handle the background chaos of actual call-centre environments.

Lost in translation.

But training on degraded data cannot overcome the bigger challenge. The dominant AI narrative claims that bigger models are inherently better.

However, research from Microsoft Research’s DELEGATE-52 benchmark shows that when input context exceeds 20 lines, standard commercial LLMs exhibit content corruption rates of up to 25%. The longer the explanation, the higher the risk of hallucination.

Furthermore, in culturally diverse regions like South Africa, customer interactions rarely stick to a single language. Even when a caller starts the conversation in English, they frequently switch to their mother tongue to explain complex scenarios or convey emotional nuance.

Traditional pipelines process speech probabilistically: the speech engine makes a best guess, passes the text to the next module, and if the initial translation fails, errors cascade through the system. As such, voice engines must recognise the precise point where a caller shifts languages. Failing to capture a mid-sentence switch results in lost context and critical misunderstandings.

Accommodating special populations and non-fluid speech.

Similar issues emerge when trying to accommodate special populations and non-fluid speech.

Standard Voice Activity Detection (VAD) assumes fluid, conversational cadence. When a caller pauses, speaks softly, or hesitates, conventional AI engines frequently interrupt with prompts, like "please repeat that", eroding user experience.

In high-stakes environments, such as a private healthcare plan provider handling calls from patients experiencing severe shortness of breath, standard VAD breaks down completely.

Finding the fix.

When Connect set out to solve this challenge, we didn’t want to make a global system slightly better at native languages or dialects. We rebuilt the whole pipeline around the callers being excluded.

This fix starts with the training data. In underserved languages, recorded audio is typically scarce, but written text exists, often in abundance. As such, an in-house text-to-speech (TTS) module can generate synthetic audio to train and teach the listener – the speech-to-text (STT) module.

This piece in the AI orchestration puzzle is one most people find genuinely surprising. While industry typically treats TTS and STT as two separate products sold to two separate buyers, we connect them.

We build a system that listens (STT) and a TTS system that turns text into a natural, cloneable human voice. In this way, the speaker teaches the listener.

The ability to take the more abundant text and have the voice engine read it aloud in hundreds of different synthetic voices, across ages, genders and regional accents teaches the listening system.

The other important training input relates to quality. We deliberately train the model on bad audio. While conventional approaches view poor line quality as an obstacle to crisp response and, as such, is filtered out and cleaned up, we embrace the crackle, the compression, the taxi rank in the background, the cheap handset.

These are not defects in the data. They are the data. Real customers do not call from recording studios. They call from noisy streets on inexpensive phones over compressed networks. That’s why we do the opposite of cleaning.

We take our training material and deliberately degrade it, running it through the same compression codecs the mobile networks use, layering in real-world noise, simulating the acoustics of small rooms and open spaces.

We also ask for those imperfect recordings sitting in a contact centre's archive, which most vendors regard as worthless, as they are the most valuable training asset. And they belong to the operator, not to a global technology company.

Armed with this real-world training data, we teach the system to navigate the world it will operate in, because a clinical proof of concept (PoC) test seldom translates into real-world operational success.

When one product manufactures the raw material for the other, and we recreate the real-world conditions, every improvement to the voice makes the ear sharper. Every hour of real customer audio makes both better.

Our competitors cannot copy this model because it only works if you own both halves. A vendor renting its speech recognition from one supplier and its voice from another has two black boxes that they cannot wire together. Owning the full stack is not vanity engineering; it’s what makes the loop possible.

Crafting fluid CX.

Trained according to this model, combining small, full-stack voice models with targeted intent extraction can deliver accurate, low-latency, and fully compliant Voice AI across every customer demographic.

The AI is trained to recognise slow or irregular speech patterns and adjusts its cadence to match the caller's rhythm and tempo. The engine can also accommodate pause disambiguation, distinguishing between the end of an utterance and a pause caused by physical difficulty or hesitation.

This is possible because, where conventional bots detect which language the customer is speaking and hand the audio to the right specialist model, "switching languages" never happens in our system.

In the real world, multilingual customers do not generally speak one language per sentence. In South Africa, for example, they can mix English with isiZulu, Afrikaans or Sesotho within a single breath - English words carrying Zulu grammar, a Sesotho phrase inside an English sentence.

A detect-and-switch system is permanently a half-second behind, forever guessing where one language ends, which means it fails most severely at exactly the moment of the switch, which for these callers is most of the conversation.

Our approach is to never ask the question. The system learns the mixed way people genuinely speak as a single language. There is no switch to detect because, as far as the model is concerned, no switch occurred.

We reinforce this by manufacturing mixed-language material at scale, generating realistic blended sentences and voicing them through our synthetic speakers, because naturally occurring recordings of code-mixed speech are among the rarest data on earth.

Furthermore, we pass on the doubt, not just the answer. A conventional speech system produces its best guess, hands over a sentence, and the next stage works from that sentence.

The flaw in this model is that the moment the system commits to one interpretation, every alternative is lost. If it mishears a policy number, nothing downstream can recover it.

Our approach is to hand the next stage a shortlist - the top candidate interpretations, along with how confident the system is in each.

A small, purpose-trained language model then chooses between them using something the ear does not have: the context of the conversation. It knows the caller is disputing a debit order. It knows which products they hold. It can therefore rule out acoustically plausible interpretations but make no sense in context.

Surfacing commercially invisible customers.

Three key benefits follow from this approach, and they are commercial rather than technical. First and foremost, this approach makes the unprofitable customer profitable to serve. And that’s not a technical footnote. It is the entire business case.

In multi-cultural countries like South Africa, the excluded caller is not a niche - they are the majority of the volume and the cost.

Automating interactions that were previously impossible to automate does not shave a margin - it changes the unit economics of the whole operation. Furthermore, compliance stops being a promise and becomes an architecture.

Because the entire system runs on ordinary processors - the kind already sitting in any corporate data centre - it can be deployed inside the client's building: No specialised hardware to procure. No customer voice data crossing a border.

The industry has spent a decade waiting for someone else to solve this problem. The assumption was always that inclusion would arrive as a by-product of scale - that once the models got big enough, everyone would eventually be heard. That has not happened, and there is no sign that it will.

The languages spoken by hundreds of millions of people remain commercially invisible to systems optimised for the markets that were profitable first. Inclusion is not a by-product. It is a design decision - and it turns out to be a defensible one.