Benchmarking report finds orchestrated SLMs outperform frontier LLMs.

Connect, an AI-enabled CX specialist has published a technical benchmarking report evaluating the deployment viability of compact Small Language Models (SLMs) across enterprise workloads.

While frontier Large Language Models (LLMs) excel at open-ended reasoning and highly fluid dialogue, Alfredo Gemma, AI Solution Director at Connect, explains that asking a single generalist model to perform intent classification, numerical extraction, contextual summarisation, and safety checks simultaneously introduces operational risks.

“As organisations move beyond experimentation and start deploying AI into operational workflows, there is growing demand for models that are smaller, faster, more cost-effective, and easier to run in controlled environments,” he continues.

This is particularly relevant in regulated sectors, where data residency, privacy, latency, explainability, and operational control can be as important as raw model capability.

To address whether compact, local models can meet stringent operational requirements in highly regulated sectors, Connect conducted the benchmark study to evaluate real-world performance using the company’s proprietary 4-billion-parameter model stack, Connect AI Elly.

The benchmark framed questions around practical deployment choices: whether compact local models are viable, how much value domain adaptation adds, and whether orchestration delivers measurable gains over a single generalist model.

According to Gemma, the findings are compelling: “The benchmark report reveals that orchestrating multiple domain-specific SLM components improves task accuracy, factual grounding, and summarisation quality by ~13 percentage points overall compared to single-model architectures, outperforming many foundation LLMs.”

Benchmark design.

The benchmark study evaluated three distinct deployment patterns across realistic enterprise tasks:

1. Baseline SLM: A single 4B-parameter model running without domain tuning or automated task routing, representing a standard, off-the-shelf compact deployment.
2. Domain-Tuned SLM: The same 4B model fine-tuned specifically on domain-specific terminology, task formats, and expected response patterns.
3. Orchestrated SLM Ensemble: A multi-agent routing architecture that decomposes complex workflows across specialised micro-models and SLMs, separating duties across a domain router, entity extractor, summariser, retriever/re-ranker, verifier model, and safety classifier.

Testing covered representative enterprise datasets across three core sectors, namely finance, healthcare and insurance.

• Finance: Evaluated on intent recognition, sentiment classification, earnings-call summarisation, conversational financial reasoning, and grounded question answering.
• Healthcare: Evaluated on biomedical named entity recognition, clinical summarisation, medical RAG groundedness, and conversational safety.
• Insurance: Evaluated on intent classification and sentiment analysis alongside retrieval-augmented generation accuracy.

Comprehensive evaluation metrics.

“Rather than relying on superficial, generic language fluency scores, the evaluation deployed a multi-dimensional metric framework to capture both linguistic precision and structural reliability that reflects realistic enterprise workloads, rather than generic academic benchmarks,” elaborates Gemma.

This included classification precision, summarisation integrity, reasoning, safety, grounding, policy, safety, and compliance.

“The results demonstrate a clear progression in system performance as architecture evolves from a single model to a specialised, orchestrated workflow,” states Gemma.

These findings echo research from Microsoft, which confirms that as an LLM's prompt length or context window grows, the risk of output quality degradation (known as context rot or the loss of "attention" precision) significantly increases.

Researchers evaluated 19 prominent models across 52 professional domains over sequential editing workflows, and found that even frontier AI models corrupt or lose an average of 25% of document content over extended, multi-turn interactions – after just 20 sequential interactions, the most advanced models corrupted a quarter of the document's content.

“Using an orchestrated ensemble dramatically reduced this risk of hallucination probability and the AI model making unsupported claims, which is a major concern for corporates from a liability standpoint,” states Gemma.

Unlocking immediate value.

The benchmark reports shows that domain tuning unlocked immediate value, raising overall cross-domain accuracy from a baseline average of 72.67% to 81.20%, which represents an 8.53 percentage-point uplift.

The largest individual gains occurred in insurance sentiment (increasing from 56.00% to 82.55%) and medical QA accuracy (rising from 81.38% to 94.34%).

The Orchestrated SLM Ensemble achieved the highest score across all evaluated configurations, reaching an overall benchmark average of 85.42% – a 12.75 percentage-point improvement over the baseline model.

“One of the most compelling results for companies looking to deploy AI into the enterprise was the uplift in targeted reliability where risk is highest,” highlights Gemma. 

Under orchestration, evidence recall rose from 60.87% to 82.00%, financial earnings summarisation increased from 27.85% to 44.20%, and clinical summarisation jumped from 25.73% to 48.25%.

In cross-domain leaderboard rankings, the Orchestrated SLM Stack outperformed leading foundation models, including ChatGPT (81.60%), Gemini (80.00%), Amazon Nova (78.00%), Mixtral-8x22B (76.40%), and Qwen3-14B (73.50%), competing directly alongside Claude (83.10%) while surpassing it in key healthcare and insurance tasks.

Deploying hybrid intelligence.

“For contact centre leaders and enterprise architects, choosing between a hosted, generalist LLM and an orchestrated SLM ensemble represents a choice between unconstrained general fluency and controlled, deterministic performance,” explains Gemma

“Deploying a hybrid intelligence approach through local, sovereign infrastructure provides high cost predictability while eliminating the ‘LLM Context Tax’ associated with public cloud models.”

This localised architecture allows organisations to maintain strict data residency and privacy compliance, while establishing a fully configurable verification pipeline that reduces hallucination risks and ensures absolute explainability.

“Rather than assuming that larger models are always the best answer, the benchmark report shows that orchestrating specialised models can achieve enterprise-grade domain performance,” continues Gemma.

“The findings support routing atomic tasks like entity extraction to Micro Language Models (MLMs) and workflow synthesis to domain-tuned SLMs. This model helps organisations avoid single-vendor lock-in, eliminate unnecessary cloud processing costs, and delivers superior engagement metrics with complete data sovereignty, delivering better returns on AI investments with far less risk,” he concludes.

.