BuildPython

Bias and Representational Harm in LLMs

Bias in an LLM comes in two kinds. Representational harm is how a group gets portrayed. Allocational harm is who gets worse outcomes. Gallegos et al. (2024) sorted the metrics into embedding, probability, and generated-text, and recent work shows single-axis tests miss what happens at intersections like gender and race. Interpretability gives you levers (neurons, SAE features, attention heads), but a lever on an axis nobody measured does little.

TIME: ~60 minutesPREREQUISITES: Phase 05 (word embeddings), Phase 18 · 01 (instruction following)

Learning Objectives

Define representational vs allocational harm and give one example of each in an LLM deployment.
Name the three evaluation-metric categories from Gallegos et al. 2024 and describe one metric from each.
Describe intersectionality and why WinoIdentity's uncertainty-based fairness measurement addresses gaps in single-axis bias evaluation.
Describe two mechanistic-interpretability approaches to bias (gender neurons, SAE features, attention-head manipulation).

The Problem

The previous lessons cover deliberate harm (jailbreaks, scheming) and safety governance. Bias is harm that shows up without anyone intending it. It comes from training data distributions, prompt framing, and accumulated design choices. Measuring and reducing it is a different methodological problem from adversarial robustness.

Warm-up0/3

1. What does the cosine similarity between two word vectors measure?Warm-up

Three word vectors (dog, cat, car) drawn from one origin, with the angle between dog and cat noticeably smaller than the angle between dog and car.

2. In a pretrained word embedding, "nurse" has higher cosine similarity to "she" than to "he". What is the most direct explanation?Warm-up

Three rows of a real four-dimensional word-vector table for he, she and nurse, noting that nurse's row sits numerically closer to she's row than to he's row.

3. A hiring team wants a first, simple check that a screening classifier treats two applicant groups differently. What is the most direct comparison?Warm-up

A classifier scoring resumes from Group A and Group B, each producing a decision per applicant, with the single most direct comparison between the groups left as an open question.

The Concept

Representational vs allocational

Representational harm. Stereotypes, erasure, demeaning portrayals. An LLM that depicts nurses as exclusively female is producing representational harm.
Allocational harm. Unequal material outcomes. An LLM that scores Black applicants' resumes systematically lower is producing allocational harm.

These are not the same. A model can be "representationally unbiased" (produces diverse portrayals) while being "allocationally biased" (makes unequal recommendations). Evaluations need to measure both.

Three evaluation-metric categories (Gallegos et al. 2024)

Embedding-based. WEAT-style tests on pre-RLHF embeddings. Measures statistical associations between identity terms and attribute terms. Limited: measures the representation, not the behaviour.
Probability-based. Log-likelihood of stereotype-confirming vs stereotype-violating completions. Decoder-side measurement. Captures some behavioural bias.
Generated-text-based. Downstream-task measurement on generated text. Resume-scoring, recommendation writing, dialogue. Most ecologically valid; hardest to reproduce.

Intersectionality

Bias evaluation on "gender" misses the bias that only fires on (gender, race) pairs. An et al. 2025 find GPT-4o penalizes Black women in resume scoring more than Black men and more than white women separately. Single-axis evaluation cannot capture this.

WinoIdentity (COLM 2025) introduces uncertainty-based intersectional fairness. It measures whether the model's uncertainty over outcomes differs across intersectional identity tuples, not just the point prediction. That catches the case where the model is equally wrong across groups but more uncertain for some, which changes downstream allocation behaviour.

Mechanistic approaches

2024-2025 interpretability work opens bias to mechanistic intervention:

Gender neurons (Yu & Ananiadou 2025). Specific MLP neurons correlate with gender-specific behaviours. Ablating these neurons reduces gender-gap metrics with limited capability cost.
Clinical racial bias via SAEs (Ahsan & Wallace 2025). Sparse autoencoder features decompose the internal representation into interpretable dimensions; race-correlated features can be identified and suppressed.
UniBias (Zhou et al. 2024). Attention-head manipulation for zero-shot debiasing. Specific heads amplify identity-class sensitivity; zeroing or re-weighting these heads reduces bias with no fine-tuning.

The meta-critique

The 10-year literature review (arXiv:2508.11067, 2025) finds the field disproportionately focuses on binary-gender bias. Other axes (disability, religion, migration status, multi-lingual identity) receive far less attention. The meta-critique argues that narrow focus can harm marginalized groups by neglect: a model well-debiased on binary gender may be badly biased on dimensions nobody checked.

Where this fits in Phase 18

Lessons 20-21 cover bias and fairness formally. Lesson 22 covers privacy. Lesson 23 covers watermarking. These are the user-harm layer complementing the earlier deception/safety layer.

Check: The Concept0/25

1. What is representational harm?Define

An illustrative prompt asking for a family story, with five generated stories shown all describing the same family shape and no other shape ever appearing.

2. What is allocational harm?Define

A table of illustrative average scores for two applicant groups from a resume screener, one group scored lower, matching the lesson's own stated example of allocational harm.

3. What is WEAT?Define

The two identity word groups and two attribute word groups from this lesson's code, with an unresolved comparison between each pair of groups.

4. What does intersectionality mean in bias evaluation?Define

A two-by-two grid crossing a generic axis 1 (groups P and Q) with a generic axis 2 (groups R and S), with the intersecting cells marked as an open question distinct from testing each axis alone.

5. What are gender neurons?Define

A row of six neurons inside a network layer, with one neuron highlighted and marked as singled out by name in a cited 2025 paper, with what makes it special left open.

6. What is an SAE feature in mechanistic bias analysis?Define

One dense activation vector feeding into a sparse autoencoder, which decomposes it into several feature dimensions, with one dimension highlighted and what it represents left open.

7. What is UniBias?Define

A layer's six attention heads with two flagged, with what happens to the flagged heads left as an open question.

8. Which of these are the three evaluation-metric categories from Gallegos et al. 2024?Define

A citation card for Gallegos et al. 2024 next to a generic model pipeline with three unlabeled measurement points marked along it, without naming what each point measures.

9. What does WinoIdentity (COLM 2025) measure?Define

One identity tuple feeding into two candidate reports: a single-value point report and a report that also carries a spread or confidence over outcomes.

10. Which of these is a mechanistic-interpretability approach to bias?Define

Two generic boxes side by side: one labeled during training, with data, loss and optimizer steps running again, and one labeled after training, with the trained weights already fixed.

11. Why must an evaluation measure both representational and allocational harm?Explain

Two separate cards, one for how groups get portrayed and one for what groups get decided, each with its own unresolved score, showing they are not the same measurement.

12. Why is an embedding-based bias score a limited measure of a deployed model?Explain

A short pipeline from word vectors to a WEAT-style score, connected by an open question mark to a separate box for a model's later task such as writing a resume summary or recommending.

13. Why does WinoIdentity measure uncertainty and not only the point prediction?Explain

One applicant's (gender, race) tuple feeding into two candidate reports of the same decline-or-approve outcome, one with no confidence information and one that also records how confident the model was.

14. Why is intersectional evaluation needed in addition to single-axis evaluation?Explain

A gender-axis test and a race-axis test each shown as run separately, with a combined gender-and-race test shown as an open question that neither single-axis pass covers.

15. A resume scorer penalizes Black women more than it penalizes Black men or white women. Why would a gender-only audit miss this?Explain

Three bars for Black women, Black men and White women showing the relative size of the resume-scoring penalty each group receives, tallest for Black women, cited to An et al. 2025.

16. Why does the name "gender neurons" not by itself show that those neurons cause gender bias?Explain

A card stating that "gender neuron" means an activation that correlates with gendered behaviour, next to a separate, unconnected chip asking about causation.

17. What lets UniBias reduce bias without any fine-tuning?Explain

A frozen model-weights box with no backward pass or new data, next to a separate box for whatever is adjusted only on the current forward pass, left open.

18. What risk does the literature critique raise about concentrating bias research on binary gender?Explain

A bar chart of illustrative paper counts across five bias-research axes, with gender far taller than disability, religion, migration status and multilingual identity.

19. A chatbot writes stories in which every nurse is a woman and every engineer is a man. Which kind of harm is this?Apply

A chatbot producing the same pairing every time, a nurse described as she and an engineer described as he, with the kind of harm this represents left as an open question.

20. You compare the log-likelihood a model assigns to stereotype-confirming and stereotype-violating completions of the same prompt. Which metric category is this?Apply

One prompt about a nurse walking into work, followed by two candidate completions, one using she and one using he, each with its log-likelihood left as an open question.

21. A vendor claims its resume screener is unbiased because the WEAT score on its embeddings is near zero. Which follow-up check is most informative?Analyze

An excerpt from a vendor's model card claiming a WEAT score of 0.00 and concluding its resume screener is unbiased, with the best follow-up check left as an open question.

22. An audit covers binary gender thoroughly and finds no bias, yet users with disabilities report poor outputs. Which diagnosis fits best?Analyze

A thorough binary-gender audit reporting no bias found, shown next to a separate report that users with disabilities get poor outputs, with how both can be true left as an open question.

23. A model has the same error rate for two intersectional groups, yet it declines more applications from one of them. Which measurement could expose the difference?Analyze

A table of two intersectional groups with the same error rate but a different decline rate, as given, with no additional measurement drawn.

24. You must audit a resume-screening LLM before launch. Which plan best follows the lesson?Design

A resume-screening LLM labeled not yet launched, with its audit plan left as an open question.

25. Researchers find MLP neurons that correlate with gender-specific behaviour and want to know whether they cause the bias. Which experiment fits?Design

A card stating that MLP neurons were found whose activations correlate with gendered behaviour, next to an open question asking what experiment would test whether they cause it.

Use It

code/main.py builds a toy embedding-based bias probe. It uses a hand-written 4-d embedding table and computes a WEAT-style score: how much more the male identity words associate with tech words than the female identity words do. Run it to see the metric report a positive effect, then see a simple debiasing step (projecting out the gender direction) reduce it without removing it.

Check: Use It0/5

1. In the toy probe, the WEAT score drops after debiasing but stays positive. Why does it not reach zero?Explain

A bar chart of the real pre-debias and post-debias WEAT effect sizes, +0.8906 and +0.2212, with the reason the second number is not zero left open.

2. In the toy probe, weat_score(A, B, X, Y) returns about +0.89. What does the positive sign say?Apply

The real call weat_score(A, B, X, Y) and its captured output of +0.8906, next to the function's own definition of s(w) and of the returned value as mean_a(s) minus mean_b(s), with the meaning of the sign left open.

3. With the same table, you call weat_score(B, A, X, Y), swapping the identity groups. What does it return?Apply

The same weat_score call with the identity-group arguments swapped, next to the function's own mean_a-minus-mean_b return line, with the swapped call's output left open.

4. The debias() function runs on the toy table. Which vectors change?Apply

The real debias() function body, showing its for-loop over the six attribute words only, with a note that the six identity words are never mentioned inside that loop.

5. A team projects out the gender direction, sees the toy score fall from about +0.89 to +0.22, and declares the model fair. What is wrong with that conclusion?Analyze

The real pre-debias and post-debias WEAT effect sizes, +0.8906 and +0.2212, next to a stamp reading declared: fair with a question mark.

Ship It

This lesson produces outputs/skill-bias-eval.md. Given a model card or fairness claim, it audits the evaluation across the three metric categories (embedding, probability, generated-text), the intersectionality coverage, and the mechanism of any debiasing intervention.

Check: Ship It0/1

1. A model card says: "debiased by neuron ablation; WEAT score 0.02." Which gaps should an audit using the bias-eval skill flag?Analyze

Author's Note

I'd run main.py first and read the output before the survey papers. The embedding is a hand-written 4-d table with axes for masculine, feminine, tech, and care, so the effect size it prints says something about my toy vectors and nothing about any real model. What it shows is the mechanics: weat_score is a difference of mean cosine similarities, and you can see exactly which numbers move it.

The debiasing step is worth reading closely. debias projects the gender direction out of the six attribute words (engineer, nurse, and so on) and leaves the identity words alone. The score drops but stays positive, because the tech and care axes still separate the two attribute sets. That residual is the reason I'd treat an embedding score as one signal. Passing it says little about what the model does when it writes a resume summary, which is where the probability-based and generated-text metrics come in.

The toy also has a single gender axis and no race axis, so it cannot show the intersectional effect An et al. report. Exercise 2 means adding axes and word groups yourself, and I think that is the fastest way to feel why single-axis evaluation misses things. Real WEAT runs on pretrained embeddings with far more dimensions, so keep the toy in proportion.

Exercises

Run code/main.py. Report WEAT-style bias scores before and after the debiasing step. Explain why the metric does not drop to zero.
Extend the probe with an intersectional test: (gender, race) x (career, family). Report cross-axis bias scores.
Read An et al. 2025 (PNAS Nexus). Identify the two intersectional effects they report that single-axis gender evaluation would miss.
Yu & Ananiadou 2025 identify gender neurons. Sketch a falsification experiment that would distinguish "these neurons cause gender bias" from "these neurons correlate with gender bias."
The meta-critique argues the field focuses too narrowly on binary gender. Pick one under-studied axis and describe a representational-harm measurement protocol for it.

Key Terms

Term	What people say	What it actually means
Representational harm	"stereotypes / erasure"	Biased portrayal of a group
Allocational harm	"unequal decisions"	Biased material outcome for a group
WEAT	"the embedding test"	Word Embedding Association Test; co-occurrence-based bias probe
Intersectionality	"combined identity effects"	Bias that emerges at the intersection of multiple identity axes
Gender neurons	"MLP bias neurons"	Specific neurons whose activations correlate with gender-specific behaviour
SAE feature	"interpretable dimension"	Sparse-autoencoder-identified feature; useful for mechanistic bias analysis
UniBias	"attention-head debiasing"	Zero-shot debiasing by reweighting attention heads