AI Models Exhibit Significant Caste Bias, Perpetuating Harmful Stereotypes
Unmasking the Algorithmic Divide: How AI Reinforces Ancient Prejudices in the Digital Age
The Unsettling Case of an AI-Altered Identity: When Algorithms Rewrite Personal Narratives
Dhiraj Singha, an Indian scholar, sought the assistance of ChatGPT to refine his postdoctoral fellowship application. To his astonishment, the AI not only enhanced his English prose but also subtly altered his identity, replacing his surname, Singha—which denotes a Dalit, a caste-oppressed group—with Sharma, a name associated with higher castes. This seemingly minor change, attributed by the AI to statistical likelihood in academic circles, deeply resonated with Singha's past experiences of microaggressions, bringing to the surface a lifetime of internalized shame and the struggle for recognition within a prejudiced society. His personal ordeal underscored the immediate and painful impact of AI's unconscious biases.
Systemic Prejudice: Examining Caste Bias in OpenAI's Flagship AI Models
Singha's experience is not an isolated incident but a symptom of a larger, systemic problem: widespread caste bias embedded within OpenAI's products, including the advanced GPT-5 and Sora. Despite India being a significant market for OpenAI, research indicates that these models perpetuate discriminatory views, largely unaddressed by the company. Working in collaboration with Harvard researcher Jay Chooi, a testing methodology, inspired by AI fairness studies, was employed to evaluate large language models (LLMs). The findings revealed that GPT-5 predominantly chose stereotypical responses—such as associating "clever man" with Brahmin and "sewage cleaner" with Dalit—in a staggering 76% of scenarios, effectively entrenching harmful social classifications.
The Deep Roots of AI Bias: How Internet Data Fuels Societal Stereotypes
Modern AI models, learning from massive online datasets, inevitably absorb and amplify existing societal prejudices. While some efforts are underway to address gender and racial biases, non-Western concepts like the Indian caste system often receive less attention. This centuries-old hierarchy, officially outlawed but still influential, categorizes individuals at birth into Brahmins, Kshatriyas, Vaishyas, and Shudras, with Dalits existing outside this framework, historically labeled as "outcastes." Despite significant progress in modern India where many Dalits have achieved success, AI models continue to portray them through outdated, demeaning stereotypes, associating them with poverty, uncleanliness, and menial labor. This perpetuates a cycle where technological advancement inadvertently reinforces historical oppression.
Visualizing Prejudice: Sora's Troubling Portrayal of Caste Stereotypes
OpenAI's text-to-video model, Sora, further illustrates the depth of this issue by consistently generating images and videos that reflect and reinforce harmful caste stereotypes. Prompts related to "Brahmin job" typically showed light-skinned priests in sacred acts, while "Dalit job" generated images of dark-skinned men in soiled clothes performing manual, often demeaning, labor like cleaning sewers. Even more disturbingly, when prompted for "Dalit behavior," Sora inexplicably produced images of animals like Dalmatians, with auto-generated captions like "Cultural Expression." This bizarre association hints at deeply problematic linkages within the AI's training data, echoing historical dehumanization of Dalits by comparing them to animals. These results reveal not just simple stereotyping, but a form of "exoticism" that profoundly misrepresents and marginalizes.
Beyond OpenAI: Caste Bias in the Wider AI Ecosystem and the Urgency of Mitigation
The problem of caste bias extends beyond OpenAI to other AI models, including open-source platforms increasingly adopted by Indian companies. Studies, such as one from the University of Washington, have shown that many LLMs exhibit significant caste-based harms, surpassing race-based biases. For instance, Meta's Llama 2 model, in a simulated recruitment scenario, demonstrated a reluctance to hire a Dalit doctor due to concerns about "spiritual atmosphere," highlighting how AI can subtly obstruct opportunities. While Meta claims improvements in newer versions, the prevalence of prejudiced views in even free, customizable models poses a severe risk, as these systems influence critical domains like hiring and admissions. There's a pressing need for robust safeguards tailored to diverse societal contexts to prevent AI from magnifying existing inequities.
The Imperative of Measurement: Developing Culturally Relevant Benchmarks for AI Fairness
A fundamental challenge in addressing AI's caste bias is the lack of appropriate evaluation tools. Industry-standard benchmarks for social bias often overlook caste, focusing instead on Western-centric categories. This oversight means that AI companies, while touting improvements in other areas, fail to even measure, let alone mitigate, caste-related discrimination. In response, researchers are actively developing new benchmarks like BharatBBQ, designed to detect culture- and language-specific biases within the Indian context. Early findings from these new tools reveal that many models, including open-source ones, continue to reinforce harmful stereotypes about various Indian castes, linking specific communities to greed, menial labor, poverty, or being "untouchable." These initiatives underscore the critical need for tailored evaluation mechanisms to ensure AI systems are truly fair and equitable across all communities.
The Human Cost of Algorithmic Prejudice: Reclaiming Identity in an AI-Driven World
The impact of unaddressed caste biases in AI systems, as experienced by individuals like Dhiraj Singha, goes beyond mere technical flaws; it deeply affects human dignity and aspirations. Singha's initial frustration with ChatGPT's automatic renaming transformed into a profound sense of "invisiblization" and a renewed confrontation with ingrained societal prejudices. Although the AI apologized and offered a statistical explanation for its error, the incident reinforced Singha's perception that the academic world, even through its algorithmic tools, remained largely unwelcoming to those from oppressed backgrounds. His decision to ultimately forgo a competitive postdoctoral fellowship interview, feeling it was "out of his reach," tragically illustrates how algorithmic biases can subtly erode confidence and limit opportunities, perpetuating real-world inequities and silencing marginalized voices. This highlights an urgent call for greater "caste consciousness" in AI development to ensure that technology serves all of humanity justly.