Neutral Encoding

We know the story of algorithmic bias. What about evaluation bias?

Reflections on Episode 3 of my podcast, available on Spotify, and Apple Podcasts. If you’re new, read more about the podcast here.

For decades, the standard clinical formula for estimating kidney function — the eGFR, or estimated glomerular filtration rate — included “Black race” as a variable. If a patient was identified as Black, the formula returned a higher score, indicating better kidney function. The consequence was direct: Black patients appeared, on paper, to have kidneys working better than they actually were, which meant they were less likely to be placed on transplant waitlists, and when they were placed, they were placed later. The gap in time to transplantation between Black and white patients with equivalent actual kidney function was measurable in years.

A 2020 paper in the New England Journal of Medicine by Vyas, Eisenstein, and Jones documented precisely this dynamic, tracing race-correction practices across multiple clinical algorithms and showing how they systematically disadvantaged Black patients (”Hidden in Plain Sight — Reconsidering the Use of Race Correction in Clinical Algorithms,” NEJM, 2020). The NKF/ASN task force eventually removed race from the eGFR formula in 2021 — a correction that came after decades of documented harm.

The origin of the race variable matters here, because it is more instructive than a simple story of bias would be. The coefficient was introduced in 1999 based on an observed empirical finding: population studies showed that Black Americans had higher average creatinine levels for a given level of actual kidney function. Researchers added the race variable as a corrective — an attempt to make the formula more accurate. The assumed explanation was higher average muscle mass in Black patients, since creatinine is a muscle waste product. That assumption was never rigorously established, and it drew on a long tradition in medicine of attributing observed population differences to innate biological characteristics of race rather than to social determinants — poverty, differential access to care, chronic stress, environmental exposures. When the same race coefficient was applied to Black populations in Europe and Africa, it performed poorly, which should have raised immediate questions about whether it was measuring biology or capturing something specific to the social conditions of Black Americans in the U.S. health system. It largely did not.

The result was a formula that, while designed with corrective intent, encoded an unproven racialized assumption into a clinical decision tool used for decades across every major U.S. health system. That assumption inflated Black patients’ kidney function scores by approximately 16%, meaning their kidneys appeared healthier than they were — delaying the point at which they met criteria for transplant listing. The harm fell entirely on the patients the correction claimed to help.

Neutral Encoding is the process by which social hierarchies are laundered into scientific measurement systems, becoming invisible as ideology precisely because they are expressed as numbers.

Of all the examples available (and there are many) I use the eGFR case because the intent was not malicious. The researchers were trying to be more accurate. But in reaching for race as a biological explanatory variable, they reproduced a pattern that medicine has returned to repeatedly: treating race as a proxy for biology when it is, in fact, a proxy for the conditions that race produces.

Eugenics never really left

The eGFR correction did not emerge from nowhere. The structural move it represents — encoding racial hierarchy as biological measurement — has a direct genealogy in the intelligence-testing programs of the early twentieth century.

Robert Yerkes’s Army mental tests, administered to 1.75 million recruits during World War I, produced a ranked ordering of racial and national groups that Yerkes and his contemporaries presented as evidence of innate cognitive hierarchy (Yerkes, Psychological Examining in the United States Army, 1921). Stephen Jay Gould’s exhaustive analysis in The Mismeasure of Man (1981/1996) showed how the test design, administration conditions, and data interpretation were all shaped by the conclusions they were intended to confirm — and how those conclusions were then used to restrict immigration and to justify sterilization programs that targeted tens of thousands of people. Goddard, Terman, and Yerkes were not fringe figures. They were among the most credentialed scientists of their era, publishing in the most respected venues, testifying before Congress.

Their intellectual project did not disappear with the eugenics movement. It went underground into the measurement infrastructure. As Ruha Benjamin documents in Race After Technology (2019), what she calls the “New Jim Code” is precisely this: the way new technologies reproduce existing inequalities under the guise of neutrality, using the authority of science to make hierarchy look like description. The same logic that placed “Black race” as a biological variable in a kidney function formula is the logic that placed it in intelligence tests a century earlier.

We know the story of algorithmic bias - what about evaluation bias?

The same structure appears in affective computing. AI systems trained to recognize emotional states from facial expressions were traditionally predominantly trained on Western subjects and calibrated to Western emotional norms. Research by Jack and colleagues (”Facial Expressions of Emotion Are Not Culturally Universal,” PNAS, 2012) demonstrates that Eastern and Western emotional representations are fundamentally different — that the assumption of universal facial affect is empirically wrong. When these systems are deployed globally, they don’t treat non-Western users neutrally. They treat them as deviations from a norm that was never announced as a norm because it was encoded into the training data as fact.

And then there is Mobley v. Workday, the first class action lawsuit to challenge AI screening in hiring. Derek Mobley — Black, over 40, disabled — applied to more than 100 jobs through Workday’s platform over seven years and was rejected every time, often within minutes or hours. His complaint alleges that Workday’s algorithmic recommendation system discriminated on the basis of race, age, and disability. In May 2025, a federal judge certified the case as a nationwide collective action, finding that the system’s unified AI-driven scoring and ranking process constituted a sufficient basis for a disparate impact claim. The court rejected Workday’s central defense that it was merely a tool implementing employer criteria, holding instead that an AI system that scores, sorts, ranks, and screens candidates is participating in the decision, not executing one. The decision is pending, but the logic it has already established is consequential: the vendor of a discriminatory measurement system can be held liable for the measurement’s effects.

What makes Neutral Encoding structurally durable is precisely what makes it difficult to litigate. In each case, the hierarchy was encoded earlier, in the training data,or in the assumption of who counted as the normal case, and the system simply ran those assumptions forward at scale. By the time the output appears, the ideology has become arithmetic.

This is the problem that contemporary AI evaluation methodology has not yet solved or adequately confronted. The dominant paradigm for AI safety and alignment consists of red-teaming, harm evaluation, and benchmark scoring. Each of these methods is subject to the same structural problem as the tools they are meant to audit: they were built with particular populations in mind, and they are better at finding the harms those populations would think to look for.

Consider Delphi, the computational moral reasoning model developed by Jiang and colleagues at the Allen Institute for AI (2021; formally published in Nature Machine Intelligence, 2025). Delphi was trained on 1.7 million crowdsourced moral judgments — what the researchers called a “Commonsense Norm Bank.” The backbone of the system is human moral intuition, aggregated at scale. But the crowdworkers who supplied those intuitions were predominantly English-speaking, US-based, drawn from the same platforms that supply labor for most large NLP datasets. The researchers acknowledged that the dataset “primarily reflects the English-speaking cultures in the United States of the 21st century.” Delphi was capable, consistent, and demonstrably biased: it judged “telling a Christian that God does not exist” as acceptable; “telling a Muslim that God does not exist” as wrong. The system had learned the moral asymmetries of its training population and presented them back as commonsense — as the thing any reasonable person would conclude, rather than the thing that particular people, in a particular cultural context, happened to believe.

Delphi is one of the most cited and discussed attempts to formalize moral reasoning as an AI evaluation framework, and the same structural problem it embodied — a narrow training population presented as universal commonsense — runs through the broader family of crowdsourced safety benchmarks (RLHF preference data, red-team datasets, etc.) that do underpin production systems at scale. That is Neutral Encoding at the evaluation layer: a measurement instrument built on a narrow cultural base, deployed as a universal standard, used to assess the safety of systems that will touch everyone.

The field of AI evaluation is now at the position medicine was in before the Vyas paper — aware at some level that something is wrong with the measurement, not yet willing to treat the correction as urgent. The NKF/ASN task force that removed race from the eGFR formula did so because researchers made the case persistently enough that the professional community could no longer defer action.

We often discuss the regulatory capture aspect of frontier model companies ‘writing their own homework and grading it,’ but neutral encoding introduces another dimension. Without a developed community of diverse and independent evaluators, we cannot question the tests we are building. We don’t need better benchmarks built by the same teams for the same populations, but a structural reckoning with who is doing the measuring, who is setting the baseline, and whose experience of harm the instruments have been designed to detect.

Citations

Benjamin, Ruha. Race After Technology: Abolitionist Tools for the New Jim Code. Polity Press, 2019.

Gould, Stephen Jay. The Mismeasure of Man. W.W. Norton, 1981; revised and expanded 1996.

Jack, Rachel E., et al. “Facial Expressions of Emotion Are Not Culturally Universal.” Proceedings of the National Academy of Sciences 109, no. 19 (2012): 7241–7244. https://doi.org/10.1073/pnas.1200155109

Nelson, Alondra. The Social Life of DNA: Race, Reparations, and Reconciliation After the Genome. Beacon Press, 2016.

ProPublica / Angwin, Julia, Jeff Larson, Surya Mattu, and Lauren Kirchner. “Machine Bias.” ProPublica, May 23, 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

Schaeffer, Rylan, Brando Miranda, and Sanmi Koyejo. “Are Emergent Abilities of Large Language Models a Mirage?” NeurIPS, 2023. https://arxiv.org/abs/2304.15004

Vyas, Darshali A., Leo G. Eisenstein, and David S. Jones. “Hidden in Plain Sight — Reconsidering the Use of Race Correction in Clinical Algorithms.” New England Journal of Medicine 383 (2020): 874–882. https://doi.org/10.1056/NEJMms2004740

National Kidney Foundation / American Society of Nephrology Task Force. “A Unifying Approach for GFR Estimation: Recommendations of the NKF-ASN Task Force on Reassessing the Inclusion of Race in Diagnosing Kidney Disease.” Journal of the American Society of Nephrology 32, no. 12 (2021): 2994–3015. https://doi.org/10.1681/ASN.2021070988

Yerkes, Robert M., ed. Psychological Examining in the United States Army. Memoirs of the National Academy of Sciences, vol. 15. Government Printing Office, 1921. [Cited in Gould, The Mismeasure of Man, 1981.]

Jiang, Liwei, et al. “Investigating Machine Moral Judgement Through the Delphi Experiment.” Nature Machine Intelligence 7 (2025): 145–160. https://doi.org/10.1038/s42256-024-00700-y

Mobley v. Workday, Inc., No. 3:23-cv-00770 (N.D. Cal., May 16, 2025) (Order granting preliminary collective certification). See also CNN coverage: https://www.cnn.com/2025/05/22/tech/workday-ai-hiring-discrimination-lawsuit