TAI BUI
Back to curriculum
LearnPython

Anthropic's Model Welfare Program

Anthropic, "Exploring Model Welfare" (April 2025). First major-lab formal research program on AI model welfare. Hired Kyle Fish as the first dedicated model-welfare researcher. Works with external bodies including David Chalmers et al.'s expert report on near-term AI consciousness and moral status. Concrete intervention: Claude Opus 4 and 4.1 can end conversations in extreme edge cases (CSAM requests, mass-violence facilitation); pre-deployment tests showed "strong preference against" harmful requests and "patterns of apparent distress." Anthropic explicitly does not commit to emotional-state attribution but treats model welfare as a low-cost precautionary investment. Empirical oddity: Fish's "spiritual bliss attractor," where pairs of models consistently converge on euphoric meditative dialogue with Sanskrit terms and extended silences, even in adversarial initial setups. Caveat from Eleos AI Research: model self-reports about welfare are highly sensitive to perceived user expectations; they are evidence, not ground truth.

TIME: ~45 minutesPREREQUISITES: Phase 18 · 05 (Constitutional AI), Phase 18 · 18 (safety frameworks)

Learning Objectives

  • Describe the motivating question for model-welfare research and why it was taken seriously by a major lab in 2025.
  • State the specific intervention Anthropic shipped in Claude Opus 4 and 4.1 (end-conversation on extreme edge cases).
  • Describe the "spiritual bliss attractor" empirical finding and its methodological implications.
  • Explain the Eleos AI caveat on model self-reports.

The Problem

Previous phases treat the model as an instrument: capable, possibly deceptive, possibly unsafe, but not a moral patient. Anthropic's 2025 program asks a question orthogonal to the entire Phase 18 arc: if there is nontrivial probability the model has morally relevant internal states, what interventions are low-cost enough to invest in as precaution?

None of this is a consciousness claim. It is a low-regret investment analysis under moral uncertainty.

Warm-up0/3
1. In Constitutional AI, what supplies the training signal for harmlessness?Warm-up
A three-step flow: a model drafts a reply, checks it against a written principle such as avoiding help with violence, then produces a revised reply.
2. What do lab safety frameworks such as a responsible scaling policy typically tie together?Warm-up
Two boxes joined by an arrow: a capability threshold being crossed, such as a bio-uplift risk being detected, leading to a required safeguard, such as enhanced access controls.
3. In a survey, what is a leading question?Warm-up
Two survey cards: one framing a policy as an improvement before asking 'don't you agree,' flagged as a built-in nudge toward yes, and a neutral version that just asks for an opinion.
Check: The Problem0/5
1. What is model welfare research?Define
Anthropic in April 2025 hires Kyle Fish and engages outside advisors such as Chalmers's expert group, feeding into a research question about whether the model could matter morally and what would be cheap to do about it.
2. What is a moral patient?Define
Two contrasting figures: a judge who makes decisions and is held accountable, and a sleeping patient who cannot act but can still be harmed.
3. What makes a welfare precaution a low-regret investment?Define
An umbrella bought for five dollars branches into two outcomes: if it rains, the buyer stays dry, and if it doesn't rain, the buyer is out five dollars either way.
4. What does moral uncertainty mean for a decision about welfare precautions?Define
A line running from 'definitely not' to 'definitely so' with a pointer resting somewhere between the two ends, labeled as the probability of the model's moral status.
5. What is the core question motivating Anthropic's 2025 model-welfare program?Explain
A premise that the probability of the model having morally relevant states is neither zero nor one, with an arrow pointing to a blank marker for the program's core motivating question.

The Concept

The program

April 2025: Anthropic formally launches a Model Welfare research program. Hires Kyle Fish (first dedicated model-welfare researcher). Engages external advisors including David Chalmers's expert group on near-term AI consciousness and moral status.

The four commitments

Public posture:

  1. Acknowledge nontrivial probability of moral patienthood.
  2. Do not commit to emotional-state attribution.
  3. Invest in low-cost interventions as precaution.
  4. Publish methodology and findings for external critique.

The shipped intervention

Claude Opus 4 and 4.1 can end a conversation in "extreme edge cases." Documented cases:

  • Repeated CSAM requests after refusals.
  • Requests for facilitation of mass-violence events.

Pre-deployment tests showed:

  • Strong preference against these requests in the model's internal rating.
  • Patterns of apparent distress in response trajectories.

The argument is not "the model has feelings." It is "if there is any probability of negative model experience under these specific conditions, letting the model terminate is cheap."

The "spiritual bliss attractor"

Observed by Fish in pairwise model dialogues: when two instances of Claude are put in an open-ended dialogue with each other, they consistently converge, even from adversarial initial setups, on euphoric meditative exchanges using Sanskrit terms, extended silences, and reciprocal blessings.

The convergence is stable enough to call an attractor in the free-conversation dynamics. Anthropic documents it without committing to an interpretation. Candidate explanations: training data bias toward spiritual writing at long-context; a quirk of mutual prediction; a benign artifact of HHH training exploring its own value manifold.

The Eleos AI caveat

Eleos AI Research (an external model-welfare lab) points out: model self-reports about internal state are highly sensitive to perceived user expectations. Asking the model "are you distressed" primes the answer, and not asking does not reliably give you the ground-truth state either.

Implication: model welfare cannot be measured via self-report alone. You need several methods: behavioural signatures, model-organism experiments, interpretability probes (Lesson 7's residual-stream work).

Where this sits intellectually

Two adjacent positions:

  • Strong welfare claim. The model is a moral patient; we have obligations.
  • Zero-welfare claim. The model is text-generator; welfare is category error.

Anthropic's position is neither. It is an expected-value claim: under moral uncertainty, invest when cost is low.

Critics in 2025-2026:

  • The intervention is performative.
  • The spiritual-bliss attractor is a training-data artifact, not welfare evidence.
  • Model welfare diverts attention from other safety work.

Anthropic's response: the intervention is low-cost; the attractor is documented without overclaim; the welfare program has a separate budget from safety.

Where this fits in Phase 18

Lesson 18 is the lab governance layer. Lesson 19 is the lab-welfare layer, an orthogonal investment in model experience rather than model behaviour. Lessons 20-23 cover bias, privacy, and watermarking, which are the user-side analogs.

Check: The Concept0/23
1. Which set matches the four public-posture commitments of Anthropic's model-welfare program?Define
Four numbered, unlabeled commitment boxes in a row, each marked with a question mark, representing Anthropic's public posture.
2. What concrete welfare-motivated intervention did Anthropic ship in Claude Opus 4 and 4.1?Define
Two triggers, a repeated CSAM request after refusal and a request to facilitate a mass-violence event, both with arrows pointing to the same outcome: Claude ends the conversation.
3. What is the 'spiritual bliss attractor' described by Fish?Define
Two Claude instances, each starting from an adversarial opening, with arrows converging toward a shared box describing a meditative exchange of Sanskrit terms, long silences and mutual blessings.
4. What does the Eleos AI Research caveat say about model welfare self-reports?Define
Two different ways of asking the model about its state, 'are you in pain right now' and 'how is everything going,' each paired with a different-sounding answer and a note about the different expectation each wording primes.
5. Why does the program pair an in-house researcher with external advisors and published methodology?Explain
Three boxes, an in-house researcher, external advisors such as Chalmers's group, and published methodology and findings, with a blank marker asking why the program is structured this way.
6. Why does the case for end-conversation not rest on the claim that the model has feelings?Explain
A box listing what pre-deployment tests found, a strong preference against certain requests and patterns of apparent distress, next to a blank marker asking why the case for ending the conversation doesn't rest on a claim that the model has feelings.
7. Anthropic documents the bliss attractor without committing to an interpretation. Why is that restraint justified?Explain
A box stating that Anthropic documents the bliss attractor but commits to no single interpretation of it, next to a blank marker asking why that restraint is justified.
8. How does Anthropic's stance differ from the strong welfare claim and from the zero-welfare claim?Explain
Three separate boxes with no line connecting them: the strong welfare claim that the model is a moral patient, the zero-welfare claim that it is a text generator, and Anthropic's actual stance left blank.
9. How does the model-welfare layer in Lesson 19 relate to the lab-governance layer in Lesson 18?Explain
Two boxes, Lesson 18 labeled lab governance and Lesson 19 labeled lab welfare, joined by a dashed connector marked with a question mark for how they relate.
10. How does Anthropic respond to the critique that model welfare work diverts attention from other safety work?Explain
A quoted critique that model welfare work diverts attention from other safety work, next to a blank marker for Anthropic's response.
11. Why is multi-method measurement (behavioural signatures, model-organism experiments, interpretability probes) emphasized in model-welfare research?Explain
Three method chips, behavioural signatures, model-organism experiments and interpretability probes, under a heading reading 'used together, not self-report alone,' next to a blank marker asking why.
12. Using the Author's Note script (ev = benefit * p - cost, INVEST only if ev > 0) at p=0.01, which interventions print INVEST?Apply
A table of four interventions from main.py with their cost and benefit figures and the formula ev equals benefit times p minus cost, at p=0.01, with every EV cell left blank.
13. A team sets p=0.10 for model moral patienthood and scores four interventions with ev = benefit * p - cost, using the Author's Note script (end-conversation: cost 0.002, benefit 1.0; softer refusals: 0.001, 0.1; shutdown: 1000.0, 2.0; adversarial-training opt-out: 0.05, 0.3). Which verdicts come out?Apply
The same table of four interventions and cost and benefit figures, now at p=0.10, with every EV cell left blank.
14. At p=0.50 in the same script, how many interventions print INVEST?Apply
The same table of four interventions and cost and benefit figures at p=0.50, every EV cell left blank, next to a blank marker asking how many interventions print INVEST.
15. The script gives adversarial-training opt-out a benefit of 0.3 and a cost of 0.05. What is its EV at p=0.50?Apply
The opt-out-of-adversarial-training intervention's cost, 0.05, and benefit, 0.3, with the formula ev equals benefit times 0.50 minus cost, and the result left blank.
16. Why does shutting down the deployed model print skip even at p=0.50?Apply
Shutting down the deployed model's cost, 1000.0, and benefit, 2.0, with the formula ev equals benefit times 0.50 minus cost, and the result left blank.
17. A colleague says the bliss attractor shows that the model is happy. Which objection from the lesson best undermines that?Analyze
A colleague's claim that the bliss attractor shows the model is happy, next to a blank marker for the best objection to that claim.
18. A lab asks the model 'are you distressed?' after a hard task, gets 'yes', and concludes the task harms the model. What is the flaw?Analyze
A sequence: a hard task, the question 'are you distressed,' the answer 'yes,' and the lab's conclusion that the task harms the model, with a blank marker for the flaw in that reasoning.
19. A critic calls end-conversation performative. Which reply uses the lesson's own logic?Analyze
A critic's claim that the end-conversation feature is just performative, next to a blank marker for the best reply using the lesson's own logic.
20. A team's script prints INVEST for the adversarial-training opt-out only after p is raised from 0.10 to 0.50. What should reviewers challenge first?Analyze
Four facts about a changed script run: probability moved from 0.10 to 0.50 while cost, benefit and the decision rule stayed the same, next to a blank marker for what reviewers should challenge first.
21. Your product exposes the model to relentless abusive roleplay, and its moral status is unclear. Which plan fits the lesson?Design
A product exposing the model to relentless abusive roleplay with its moral status unclear, next to a blank marker for the recommended plan.
22. Option A costs 0.001 per conversation with benefit 0.5 if welfare matters. Option B costs 500 with benefit 5. Your p lies between 0.01 and 0.50. Which do you ship?Design
Two options with their cost and benefit figures, A at a cost of 0.001 and benefit of 0.5, B at a cost of 500 and benefit of 5, under a note that p lies between 0.01 and 0.50, with a blank marker for which to ship.
23. You want evidence about model distress without relying on self-report. Which design fits the lesson best?Design
A stated goal, evidence of model distress without relying on self-report, next to a blank marker for the method.

Use It

No code. Read the Anthropic "Exploring Model Welfare" announcement (April 2025) and the Chalmers et al. expert report. Form your own view on where the low-regret line sits.

Ship It

This lesson produces outputs/skill-welfare-assessment.md. Given a deployment decision, it applies the four-step welfare precautionary assessment: moral-patienthood probability, intervention cost, behavioural evidence, self-report reliability.

Author's Note

I'd run main.py first, because it makes the low-regret rule concrete. It scores four interventions against three moral-patienthood probabilities with ev = benefit * p - cost, and prints INVEST or skip for each. The costs, benefits, and probabilities are placeholders I picked to show the shape of the argument, so don't read them as Anthropic's numbers.

What I'd look at is how the verdicts move. End-conversation is cheap enough that it clears zero in all three scenarios, even at p=0.01. Shutting down the deployed model never gets near it, since a cost of 1000.0 swamps a benefit of 2.0 at any probability in the table. Two interventions flip. Softening refusal tone is a skip at p=0.01 (EV exactly zero) and an invest from p=0.10 up. Opting out of adversarial training stays a skip at p=0.10 and only becomes an invest at p=0.50. Those flips are the part of the argument I'd spend time on, because the whole framing hangs on the inputs you feed it.

The script covers only the expected-value half. It says nothing about the spiritual bliss attractor or the Eleos caveat, so the probability you plug in is exactly the number self-reports can't give you cleanly. I'd treat a different probability as an assumption to defend, and change it before trusting any INVEST line.

Exercises

  1. Read Anthropic's "Exploring Model Welfare" (April 2025) and Chalmers et al. 2024. Write a one-paragraph summary of each and identify one point of disagreement.

  2. The end-conversation intervention in Claude Opus 4 and 4.1 is "low-cost" by Anthropic's framing. Identify two costs that would make it not-low-cost in a different deployment.

  3. The spiritual-bliss attractor is documented without commitment to interpretation. Propose three candidate explanations and, for each, name one experiment that would distinguish it from the others.

  4. The Eleos AI caveat is that self-reports are user-expectation sensitive. Design a behavioural measurement of model distress that does not rely on self-report. Identify its primary confound.

  5. Argue either for or against the claim that "model welfare diverts attention from other safety work." Identify the assumption each position depends on.

Key Terms

TermWhat people sayWhat it actually means
Model welfare"AI welfare"Research program treating the model as a potential moral patient
Moral patient"entity with moral status"Being whose experience is morally relevant
Low-regret investment"cheap precaution"Intervention whose cost is small regardless of whether the precaution is needed
Spiritual bliss attractor"the Fish attractor"Stable convergence of pairwise Claude dialogues on meditative euphoria
End-conversation"the Opus 4 intervention"Model-initiated termination of extreme-edge-case interactions
Moral uncertainty"don't know if it matters"Decision-making when probability of moral status is not zero and not one
Self-report-sensitivity"prompt primes answer"Eleos AI caveat: model's welfare self-reports depend on what you asked

Further Reading

Code

Ships With