The future of multilingual CX is already here!

With language localisation emerging as a key AI-powered competitive differentiator for customer experience, the technology is rapidly evolving from the text level to voice-to-voice capabilities — where translation happens in real time, without ever converting speech fully into text.

By Alfredo Gemma, AI Solutions Director at Connect

Connect is spearheading this industry advancement, with our research and development in this area based on the evolution of Efficient Monotonic Multi-Head Attention (EMMA) - a cutting-edge adaptation of the transformer architecture originally developed for streaming speech tasks.

Real-time, low-latency translation.

In simple terms, EMMA allows models to process speech as it arrives — one slice of audio at a time — without waiting for the speaker to finish.  

This makes it ideal for real-time, low-latency translation, where every millisecond counts, such as emergency response services, fraud and security services, identity verification, medical triage, and technical support. 

Latency also kills conversational flow, eroding CX, with the potential to turn complaints into frustrating situations for both the agent and customer.   

The ability to translate simultaneously while the customer is speaking (or yelling) allows the agent to start formulating a response or solution before the customer pauses. This creates a seamless handover where the agent answers immediately when the customer takes a breath, making the customer feel truly heard rather than simply processed. 

For premium personalised services, like private banking or VIP hospitality, the awkward silences that characterise traditional translation can feel robotic and impersonal. Zero-latency translation preserves the rhythm of polite conversation, allowing for back-channelling that builds trust.

With our EMMA approach, we can build systems that:

• Listen, understand, and speak simultaneously in another language.
• Preserve speaker identity, emotional tone, and rhythm.
• Adapt to environmental context, such as background noise or local acoustic characteristics.

This isn’t just translation; it’s true voice localisation. A customer in Tokyo can speak to an agent in Madrid, both in their native languages, yet the conversation flows as if they shared one.

Localisation at every layer.

The challenge in realising this transformative CX capability is that you can’t patch true localisation on top of a global model. It must be embedded at every level — automatic speech recognition (ASR), language modelling, and text-to-speech (TTS).

Our pipeline ensures that each layer is tuned for the local reality it serves:

• ASR: Learns local acoustic environments, idioms, and accent variations.
• Language Model: Extracts meaning while respecting regional syntax and expressions.
• TTS: Responds in a locally adapted voice that sounds authentic, not synthetic.

The edge of what’s next.

We’re currently exploring how these streaming architectures integrate with contextual reinforcement learning, where systems adapt their translation and interpretation dynamically based on conversational outcomes. 

Another research area is localised phoneme mapping, where models learn to blend pronunciation and tone contours across languages - allowing smoother, more natural transitions in voice-to-voice translation. 

Perhaps the most ambitious goal is fully context-preserving multilingual memory, enabling AI agents to recall prior interactions across languages without losing nuance or emotional continuity. 

Imagine an AI engine that remembers a customer’s past conversation in Portuguese and continues it in French, perfectly aligned in meaning and tone.

The end of the black box.

The current big tech monolithic speech systems are powerful but opaque. They’re built for scale, not specificity. They can’t explain their errors, nor adapt quickly to niche markets. 

The future lies in transparent, adaptive architectures — systems that evolve continuously with the data and environment they serve. Hyperlocal AI that learns, speaks, and thinks like the people it interacts with. 

That’s not just localisation. It’s the beginning of contextual intelligence — AI that truly understands the world, one voice at a time.