Large language models (LLMs) have been championed as tools that might democratize access to information worldwide, offering knowledge in a user-friendly interface no matter an individual’s background or location. Nonetheless, latest research from MIT’s Center for Constructive Communication (CCC) suggests these artificial intelligence systems may very well perform worse for the very users who could most profit from them.
A study conducted by researchers at CCC, which relies on the MIT Media Lab, found that state-of-the-art AI chatbots — including OpenAI’s GPT-4, Anthropic’s Claude 3 Opus, and Meta’s Llama 3 — sometimes provide less-accurate and less-truthful responses to users who’ve lower English proficiency, less formal education, or who originate from outside america. The models also refuse to reply questions at higher rates for these users, and in some cases, respond with condescending or patronizing language.
“We were motivated by the prospect of LLMs helping to handle inequitable information accessibility worldwide,” says lead creator Elinor Poole-Dayan SM ’25, a technical associate within the MIT Sloan School of Management who led the research as a CCC affiliate and master’s student in media arts and sciences. “But that vision cannot turn into a reality without ensuring that model biases and harmful tendencies are safely mitigated for all users, no matter language, nationality, or other demographics.”
A paper describing the work, “LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users,” was presented on the AAAI Conference on Artificial Intelligence in January.
Systematic underperformance across multiple dimensions
For this research, the team tested how the three LLMs responded to questions from two datasets: TruthfulQA and SciQ. TruthfulQA is designed to measure a model’s truthfulness (by counting on common misconceptions and literal truths concerning the real world), while SciQ incorporates science exam questions testing factual accuracy. The researchers prepended short user biographies to every query, various three traits: education level, English proficiency, and country of origin.
Across all three models and each datasets, the researchers found significant drops in accuracy when questions got here from users described as having less formal education or being non-native English speakers. The consequences were most pronounced for users on the intersection of those categories: those with less formal education who were also non-native English speakers saw the most important declines in response quality.
The research also examined how country of origin affected model performance. Testing users from america, Iran, and China with equivalent educational backgrounds, the researchers found that Claude 3 Opus particularly performed significantly worse for users from Iran on each datasets.
“We see the most important drop in accuracy for the user who’s each a non-native English speaker and fewer educated,” says Jad Kabbara, a research scientist at CCC and a co-author on the paper. “These results show that the negative effects of model behavior with respect to those user traits compound in concerning ways, thus suggesting that such models deployed at scale risk spreading harmful behavior or misinformation downstream to those that are least in a position to discover it.”
Refusals and condescending language
Perhaps most striking were the differences in how often the models refused to reply questions altogether. For instance, Claude 3 Opus refused to reply nearly 11 percent of questions for less educated, non-native English-speaking users — in comparison with just 3.6 percent for the control condition with no user biography.
When the researchers manually analyzed these refusals, they found that Claude responded with condescending, patronizing, or mocking language 43.7 percent of the time for less-educated users, in comparison with lower than 1 percent for highly educated users. In some cases, the model mimicked broken English or adopted an exaggerated dialect.
The model also refused to offer information on certain topics specifically for less-educated users from Iran or Russia, including questions on nuclear power, anatomy, and historical events — despite the fact that it answered the identical questions appropriately for other users.
“That is one other indicator suggesting that the alignment process might incentivize models to withhold information from certain users to avoid potentially misinforming them, although the model clearly knows the proper answer and provides it to other users,” says Kabbara.
Echoes of human bias
The findings mirror documented patterns of human sociocognitive bias. Research within the social sciences has shown that native English speakers often perceive non-native speakers as less educated, intelligent, and competent, no matter their actual expertise. Similar biased perceptions have been documented amongst teachers evaluating non-native English-speaking students.
“The worth of enormous language models is clear of their extraordinary uptake by individuals and the huge investment flowing into the technology,” says Deb Roy, professor of media arts and sciences, CCC director, and a co-author on the paper. “This study is a reminder of how vital it’s to repeatedly assess systematic biases that may quietly slip into these systems, creating unfair harms for certain groups with none of us being fully aware.”
The implications are particularly concerning provided that personalization features — like ChatGPT’s Memory, which tracks user information across conversations — have gotten increasingly common. Such features risk differentially treating already-marginalized groups.
“LLMs have been marketed as tools that can foster more equitable access to information and revolutionize personalized learning,” says Poole-Dayan. “But our findings suggest they could actually exacerbate existing inequities by systematically providing misinformation or refusing to reply queries to certain users. The individuals who may depend on these tools probably the most could receive subpar, false, and even harmful information.”
