Saved benchmark

Voice Notes batch results

32 recordings from data/Voice Notes.xlsx. Ranking uses WER, but the decision view also checks entity recall, hallucination, latency, and failures.

Best WER

Sarvam Saarika v2.5

16.4% WER

Best entity recall

Sarvam Saarika v2.5

76.7%

Fastest response

Sarvam Saarika v2.5

815 ms avg

No-failure pick

OpenAI GPT-4o Mini Transcribe

32/32 samples

Batch benchmark table

Rank	Model	Samples	WER	Accuracy	Exact match	CER	Entity recall	Hallucination	RTF	Latency	Status
1	Sarvam Saarika v2.5	30/32	16.4%	83.6%	46.7%	5.2%	76.7%	2.8%	0.17x	815 ms	2 failed
2	Sarvam Saaras v3 Codemix	30/32	17.9%	82.1%	36.7%	4.6%	65%	2.3%	0.18x	831 ms	2 failed
3	OpenAI GPT-4o Mini Transcribe	32/32	18.4%	81.7%	34.4%	7.5%	60.4%	2.6%	0.25x	1299 ms	OK
4	Sarvam Saaras v3	30/32	18.7%	81.3%	33.3%	4.5%	65%	2.7%	0.18x	902 ms	2 failed
5	Deepgram Nova-3	32/32	24.2%	75.8%	15.6%	9.1%	35.7%	4.2%	0.33x	1731 ms	OK
6	Deepgram Nova-2	32/32	24.3%	75.7%	18.8%	9.8%	34.5%	4%	0.3x	1542 ms	OK
7	AssemblyAI Best	32/32	27.3%	72.7%	31.3%	12.4%	52.1%	4.1%	0.87x	4340 ms	OK
8	Deepgram Base	32/32	28.5%	71.5%	12.5%	19.1%	17.2%	6.7%	0.29x	1499 ms	OK
9	OpenAI GPT-4o Transcribe	32/32	29.2%	70.8%	31.3%	15.8%	57%	3.8%	0.28x	1452 ms	OK
10	Google STT Latest Long	32/32	36.6%	63.5%	0%	16.7%	0%	1.4%	0.39x	2039 ms	OK
11	Google STT Telephony	32/32	36.6%	63.5%	0%	17.2%	0%	1.3%	0.4x	2082 ms	OK

Note: Sarvam synchronous API rejected two audio files above 30 seconds; those are counted as failures for Sarvam models.

Accuracy

Higher is better

Sarvam Saarika v2.583.6%

Sarvam Saaras v3 Codemix82.1%

OpenAI GPT-4o Mini Transcribe81.7%

Sarvam Saaras v381.3%

Deepgram Nova-375.8%

Deepgram Nova-275.7%

AssemblyAI Best72.7%

Deepgram Base71.5%

OpenAI GPT-4o Transcribe70.8%

Google STT Latest Long63.5%

Google STT Telephony63.5%

WER

Lower is better

Sarvam Saarika v2.516.4%

Sarvam Saaras v3 Codemix17.9%

OpenAI GPT-4o Mini Transcribe18.4%

Sarvam Saaras v318.7%

Deepgram Nova-324.2%

Deepgram Nova-224.3%

AssemblyAI Best27.3%

Deepgram Base28.5%

OpenAI GPT-4o Transcribe29.2%

Google STT Latest Long36.6%

Google STT Telephony36.6%

Entity recall

Names and locality phrases

Sarvam Saarika v2.576.7%

Sarvam Saaras v3 Codemix65%

Sarvam Saaras v365%

OpenAI GPT-4o Mini Transcribe60.4%

OpenAI GPT-4o Transcribe57%

AssemblyAI Best52.1%

Deepgram Nova-335.7%

Deepgram Nova-234.5%

Deepgram Base17.2%

Google STT Latest Long0%

Google STT Telephony0%

Hallucination

Inserted unsupported words

Google STT Telephony1.3%

Google STT Latest Long1.4%

Sarvam Saaras v3 Codemix2.3%

OpenAI GPT-4o Mini Transcribe2.6%

Sarvam Saaras v32.7%

Sarvam Saarika v2.52.8%

OpenAI GPT-4o Transcribe3.8%

Deepgram Nova-24%

AssemblyAI Best4.1%

Deepgram Nova-34.2%

Deepgram Base6.7%

Latency

Average response time

Sarvam Saarika v2.5815 ms

Sarvam Saaras v3 Codemix831 ms

Sarvam Saaras v3902 ms

OpenAI GPT-4o Mini Transcribe1299 ms

OpenAI GPT-4o Transcribe1452 ms

Deepgram Base1499 ms

Deepgram Nova-21542 ms

Deepgram Nova-31731 ms

Google STT Latest Long2039 ms

Google STT Telephony2082 ms

AssemblyAI Best4340 ms

Real-time factor

Below 1x is faster than audio

Sarvam Saarika v2.50.17x

Sarvam Saaras v3 Codemix0.18x

Sarvam Saaras v30.18x

OpenAI GPT-4o Mini Transcribe0.25x

OpenAI GPT-4o Transcribe0.28x

Deepgram Base0.29x

Deepgram Nova-20.3x

Deepgram Nova-30.33x

Google STT Latest Long0.39x

Google STT Telephony0.4x

AssemblyAI Best0.87x

Transcript evidence

Ground truth vs model output

Showing representative samples with every model output so the numbers stay explainable.

Male_NormalMic6_Englsih_ngedhu.ogg

English / Phone Mic / 4.86s

Electronic City

Ground truth

Electronic City feels like a different town altogether

Deepgram Nova-30% WER

Electronic City feels like a different town altogether.

Entity 100%Hallucination 0%RTF 0.3x

Deepgram Nova-20% WER

Electronic City feels like a different town altogether.

Entity 100%Hallucination 0%RTF 0.35x

Deepgram Base0% WER

Electronic city feels like a different town altogether.

Entity 100%Hallucination 0%RTF 0.3x

Sarvam Saaras v3 Codemix0% WER

electronic city feels like a different town altogether.

Entity 100%Hallucination 0%RTF 0.15x

Sarvam Saarika v2.50% WER

Electronic city feels like a different town altogether.

Entity 100%Hallucination 0%RTF 0.15x

OpenAI GPT-4o Mini Transcribe0% WER

Electronic City feels like a different town altogether.

Entity 100%Hallucination 0%RTF 0.22x

AssemblyAI Best0% WER

Electronic city feels like a different town altogether.

Entity 100%Hallucination 0%RTF 0.84x

Sarvam Saaras v325% WER

Electronic City Feels Like A Different Town All Together

Entity 100%Hallucination 11.1%RTF 0.18x

OpenAI GPT-4o Transcribe100% WER

इलेक्ट्रॉनिक सिटी फील्स लाइक अ डिफरेंट टाउन altogether.

Entity 0%Hallucination 5%RTF 0.29x

Google STT Latest Long112.5% WER

इलेक्ट्रॉनिक सिटी फील्स लाइक ए डिफरेंट टाउन ऑल टुगेदर।

Entity 0%Hallucination 4.3%RTF 0.58x

Google STT Telephony112.5% WER

इलेक्ट्रॉनिक सिटी फील्स लाइक ए डिफरेंट टाउन ऑल टुगेदर।

Entity 0%Hallucination 4.3%RTF 0.52x

Male_NormalMic7_Englsih_qf0w9q.ogg

English / Phone Mic / 5.2s

Sarjapur,

Ground truth

We move to Sarjapur to be closer to the international schools

Sarvam Saarika v2.50% WER

We move to Sarjapur to be closer to the international schools.

Entity 100%Hallucination 0%RTF 0.15x

Deepgram Nova-39.1% WER

We moved to Sarjapur to be closer to the international schools.

Entity 100%Hallucination 0%RTF 0.21x

Deepgram Nova-29.1% WER

We move to Sarajapur to be closer to the international schools.

Entity 0%Hallucination 0%RTF 0.22x

Sarvam Saaras v39.1% WER

We move to Sharjahpur to be closer to the international schools.

Entity 0%Hallucination 0%RTF 0.22x

Sarvam Saaras v3 Codemix9.1% WER

We move to Sharjahpur to be closer to the international schools.

Entity 0%Hallucination 0%RTF 0.14x

OpenAI GPT-4o Mini Transcribe9.1% WER

We moved to Sarjapur to be closer to the international schools.

Entity 100%Hallucination 0%RTF 0.2x

AssemblyAI Best9.1% WER

We moved to Sarjapur to be closer to the international schools.

Entity 100%Hallucination 0%RTF 0.81x

Deepgram Base18.2% WER

We move to Sa port to be closer to the international schools.

Entity 0%Hallucination 8.3%RTF 0.23x

OpenAI GPT-4o Transcribe54.5% WER

वी moved to सर्जापुर to be closer to the international schools.

Entity 0%Hallucination 21.4%RTF 0.26x

Google STT Latest Long109.1% WER

सी मूव टू शारजाहपुर, तू भी क्लोज़र टू डी इंटरनेशनल स्कूल।

Entity 0%Hallucination 4.5%RTF 0.51x

Google STT Telephony109.1% WER

वे मूव टू सरजापुर तो बे क्लोजर टू द इंटरनेशनल स्कूल।

Entity 0%Hallucination 5%RTF 0.52x

Male_NormalMic5_Noisy_English_m7okw7.ogg

English / Phone Mic / 2.74s

Koramangala

Ground truth

I live in Koramangala

Sarvam Saarika v2.50% WER

I live in Koramangala.

Entity 100%Hallucination 0%RTF 0.3x

OpenAI GPT-4o Transcribe0% WER

I live in Koramangala.

Entity 100%Hallucination 0%RTF 0.45x

OpenAI GPT-4o Mini Transcribe0% WER

I live in Koramangala.

Entity 100%Hallucination 0%RTF 0.38x

Deepgram Nova-325% WER

I live in Kormangla

Entity 0%Hallucination 0%RTF 0.62x

Deepgram Nova-225% WER

I live in Gorhamangla.

Entity 0%Hallucination 0%RTF 0.57x

Deepgram Base25% WER

I live in Core.

Entity 0%Hallucination 0%RTF 0.5x

AssemblyAI Best25% WER

I live in Kormangala.

Entity 0%Hallucination 0%RTF 1.62x

Google STT Latest Long25% WER

No transcript returned

Entity 0%Hallucination 0%RTF 0.6x

Google STT Telephony25% WER

No transcript returned

Entity 0%Hallucination 0%RTF 0.61x

Sarvam Saaras v3100% WER

Eleven Core Mangala

Entity 0%Hallucination 0%RTF 0.23x

Sarvam Saaras v3 Codemix100% WER

Eleven Core Mangala

Entity 0%Hallucination 0%RTF 0.31x

Male_NormalMic5_Englsih_kbzq9e.ogg

English / Phone Mic / 5.38s

Jayanagar,

Ground truth

I grew up in the quite lanes of Jayanagar,

Sarvam Saaras v311.1% WER

I grew up in the quiet lanes of Jayanagar

Entity 100%Hallucination 0%RTF 0.15x

Sarvam Saaras v3 Codemix11.1% WER