Researchers from Penn State and Oregon State University discovered that laypersons fail to recognise systematic racial bias in AI training data, even when the correlation between race and emotion is explicitly visible.
The study, published in Media Psychology, examined whether people understand that unrepresentative training data leads to biased AI performance. Across three experiments with 769 participants, researchers presented 12 versions of a prototype facial expression detection system trained on racially skewed datasets. Happy faces were predominantly white, while sad faces were predominantly Black.
Most participants indicated they noticed no bias in the training data. Only when the AI demonstrated biased performance—misclassifying emotions for Black individuals whilst accurately classifying white individuals—did some participants suspect problems.
“We were surprised that people failed to recognise that race and emotion were confounded, that one race was more likely than others to represent a given emotion in the training data — even when it was staring them in the face,” said S. Shyam Sundar, Evan Pugh University Professor and director of the Center for Socially Responsible Artificial Intelligence at Penn State. “For me, that’s the most important discovery of the study.”
Identifying racial bias
Black participants proved more likely to identify racial bias, particularly when training data over-represented their own group for negative emotions.
Lead author Cheng Chen, an assistant professor of emerging media and technology at Oregon State University, said bias in performance proves “very, very persuasive,” with people ignoring training data characteristics to form perceptions based on biased outcomes.
The research suggests humans trust AI to remain neutral even when evidence indicates otherwise. The scholars, who have studied this issue for five years, said AI systems should “work for everyone” and produce outcomes that are diverse and representative.
Future research will focus on developing better methods to communicate inherent AI bias to users, developers and policymakers, with plans to improve media and AI literacy.
