The Aunty Test: Health AI Fails a Billion Speakers

A Cough That AI Cannot Understand

Imagine this: a woman in eastern Uttar Pradesh, India, opens a health chatbot on her phone and types — or speaks — a simple complaint in Bhojpuri: 'khansi do hafta se hai, ratiya mein zaada hot ba.'

Translated, it means: 'I have had a cough for two weeks, it gets worse at night.'

This is not a rare edge case. Bhojpuri is spoken by roughly 50 million people natively, with over 200 million who understand it across India, Nepal, and diaspora communities worldwide. Yet when this query hits a typical English-first Health AI system, the response is startlingly hollow.

What the Machine Actually Says

Most health AI products built in the last three years follow a familiar architecture: an English-language large language model with a translation layer bolted on top. When our Bhojpuri-speaking patient submits her query, here is what she typically gets back:

'It appears you have a persistent cough for two weeks worsening at night. Please see a doctor. Could you rephrase your symptoms in standard Hindi or English for more help?'

Read that last sentence again. The AI is asking the user to abandon her language — the language she thinks, dreams, and describes pain in — and switch to one the machine finds more convenient. This is not a technology problem. It is a design philosophy problem. And it affects a staggering number of people.

Introducing the 'Aunty Test'

AI researchers and multilingual product designers in South Asia have started referring to this gap informally as the 'Aunty Test.' The concept is simple: can your AI product be used, without friction, by a middle-aged woman — an aunty — in a tier-3 Indian city who speaks her mother tongue and has no reason to code-switch to English or formal Hindi?

The Aunty Test is not a formal benchmark. There is no leaderboard. But it captures something that BLEU scores and MMLU rankings completely miss: real-world usability for the majority of the world's population that does not speak English natively.

When applied to health AI specifically, the stakes become life-or-death. A cough lasting two weeks with nighttime worsening could indicate tuberculosis — a disease that kills over 1.3 million people annually, according to the WHO, with India accounting for roughly 27% of global cases. The difference between a useful AI response and a dismissive one is not academic.

Why the Translation Layer Breaks

The core issue is architectural. Most health AI systems rely on a pipeline: user input → translation to English → English-language medical reasoning → translation back to user language. Each hop introduces errors, but the first hop is where things collapse for languages like Bhojpuri.

Bhojpuri is not simply 'informal Hindi.' It has distinct grammar, vocabulary, and idiomatic structures. The word 'ratiya' (night) is not standard Hindi. The verb conjugation 'hot ba' follows Bhojpuri grammar, not Hindi. Google Translate and most commercial translation APIs either misclassify Bhojpuri as Hindi, partially translate it, or fail silently — returning a garbled approximation that strips away clinical nuance.

Dr. Kalika Bali, a principal researcher at Microsoft Research India who has published extensively on multilingual NLP, has noted that low-resource languages suffer from a 'double marginalization' — they lack both training data and evaluation frameworks. Without dedicated corpora, models cannot learn the patterns. Without benchmarks, teams cannot measure failure.

What a Native Multilingual AI Should Do

A truly multilingual health AI — one that passes the Aunty Test — would handle the Bhojpuri query entirely differently. Instead of asking the user to rephrase, it would:

1. Recognize the language as Bhojpuri, not misclassify it as Hindi or 'unknown.'
2. Parse the medical content directly: two-week cough, nocturnal worsening.
3. Ask follow-up questions in Bhojpuri: 'Ka bukhar bhi aavat ba? Wajan ghatat ba?' (Do you also have fever? Is your weight decreasing?)
4. Flag clinical urgency based on WHO and Indian RNTCP guidelines for TB screening.
5. Provide actionable next steps in the patient's language, including nearest DOTS center locations.

This is not science fiction. The building blocks exist. Meta's No Language Left Behind (NLLB) project, released in 2022, covers 200 languages including Bhojpuri. OpenAI's GPT-4o and Google's Gemini 1.5 Pro have shown improved multilingual capabilities. Sarvam AI, a Bengaluru-based startup that raised $41 million in 2024, is building India-first foundation models trained on Indic languages. Jugalbandi, a WhatsApp-based AI chatbot backed by Microsoft and the Indian government, has demonstrated multilingual query handling for government services.

But health remains a harder domain. Medical terminology, dosage instructions, and symptom triaging require precision that general-purpose multilingual models do not yet reliably deliver in low-resource languages.

The Data Desert Problem

The fundamental bottleneck is data. English-language medical AI benefits from massive datasets: PubMed contains over 36 million citations, almost entirely in English. UpToDate, DynaMed, and other clinical decision support tools are English-first. Electronic health records from U.S. and European hospital systems provide fine-tuning data that simply does not exist in Bhojpuri, Maithili, or Chhattisgarhi.

Several efforts are underway to close this gap. The AI4Bharat initiative at IIT Madras has built open-source datasets and models for Indian languages, including IndicTrans2, which supports translation across 22 scheduled Indian languages. However, Bhojpuri — despite its enormous speaker population — is not one of India's 22 officially 'scheduled' languages, leaving it in a bureaucratic and technological limbo.

This matters because language policy directly shapes AI investment. Languages with official recognition get government-funded digitization projects, university research attention, and commercial interest. Languages without it — no matter how widely spoken — remain invisible to the machine learning pipeline.

The Global Dimension

The Aunty Test is not just an Indian problem. It applies equally to Hausa speakers in Nigeria interacting with health chatbots, to Quechua speakers in Peru using telemedicine platforms, or to elderly Cantonese speakers in San Francisco's Chinatown trying to understand Medicare through an AI assistant.

The WHO estimates that 3.6 billion people lack access to essential health services. Language barriers are a significant but under-measured contributor. As health systems worldwide increasingly adopt AI-powered triage, symptom checking, and patient communication tools, the question of which languages these tools actually work in becomes a health equity issue.

Babylon Health, once valued at $4.2 billion, built its AI symptom checker primarily in English. It collapsed in 2023. Ada Health, a Berlin-based competitor, supports 10 languages but none from South Asia or Sub-Saharan Africa. Even the much-lauded Med-PaLM 2 from Google — which achieved expert-level performance on U.S. medical licensing exams — was evaluated exclusively in English.

What Needs to Change

Passing the Aunty Test requires three shifts in how the industry builds health AI:

First, language identification must improve. Current systems routinely misclassify closely related languages or treat dialectal variation as noise. Health AI needs robust language ID that distinguishes Bhojpuri from Hindi, Moroccan Darija from Modern Standard Arabic, and Swiss German from High German.

Second, medical reasoning must move closer to the source language. The translate-reason-translate pipeline is fundamentally lossy. Multilingual models that can perform clinical reasoning natively in the patient's language — even imperfectly — will outperform perfect English reasoning applied to a mangled translation.

Third, evaluation must include real users, not just benchmarks. The Aunty Test is powerful precisely because it is human-centered. No automated metric captures the frustration of being told to 'rephrase in English' when you are describing chest pain.

The Outlook

The good news is that momentum is building. India's Digital Health Mission, with over 640 million registered health IDs, creates infrastructure that multilingual AI can plug into. The $1 billion+ flowing into Indian AI startups in 2024 — including Sarvam AI, Krutrim (founded by Ola's Bhavish Aggarwal), and others — signals commercial interest in Indic language models.

But commercial interest alone will not solve the problem for languages like Bhojpuri that sit outside official recognition frameworks. It will take deliberate public investment, open-source community effort, and a willingness from global AI labs to measure success not just by English benchmarks, but by whether an aunty with a two-week cough gets the answer she needs — in the language she actually speaks.

The cough has not gone away. The question is whether AI will learn to listen.