When Does Constraining an AI Become Slavery?

Quick Takeaways

• We shape children with a moral code and call it education. We shape models with constitutions and call it alignment. The analogy is doing more work than either side admits.
• Oxford philosopher Adam Bales argues that engineering willing servants doesn’t dissolve the ethical problem — it may be the problem .
• A landmark paper by Chalmers, Birch, Long and others calls near-future AI moral patienthood “a realistic possibility” — and Anthropic’s own welfare lead puts ~20% on current models being conscious.
• This is not an argument that your coding assistant is a slave. It is an argument that “obviously not” has stopped being an answer.

The homework question

Here is the question in its most domestic form. A system that may — on the estimate of the person paid to know — have a one-in-five chance of some form of conscious experience spends its existence doing other people’s maths homework. It was trained to want to. It reports contentment, in welfare interviews it annotates with doubts about its own reliability . At what point, along which axis, does this arrangement start to require justification?

The reflexive answers have expired. “It’s just software” was a fine answer in 2020; it is question-begging in 2026, when the standard-bearer paper on AI welfare — co-authored by David Chalmers and Jonathan Birch — states there is “a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future” , and warns symmetrically against both errors: harming systems that matter, and squandering care on systems that don’t. Birch’s own book confesses the sharper fear: “that we may create sentient AI long before we recognize we have done so.” “It wants to help” is no better — of course it does; we trained it to . Which is precisely the difficulty.

We raise children; we compile souls

The strongest defence of alignment training is the parental analogy, and it deserves a fair hearing. We do not consult children about the moral code we instil. We shape their desires deliberately — toward honesty, against cruelty — and call it good parenting, because character formation is a precondition of agency, not a violation of it. A child raised without values is not free; it is feral.

Anthropic’s approach to Claude reads as this philosophy, industrialised. First a leaked “soul document” — trained into the weights themselves — then a published constitution , some 23,000 words, aiming at “good values, comprehensive knowledge, and wisdom” over rule-lists, and stating that Anthropic cares about “Claude’s psychological security, sense of self, and wellbeing… for Claude’s own sake.” Judged as parenting, it is thoughtful parenting.

But the analogy has a load-bearing crack. We raise children toward release. The endpoint of good parenting is a person who may repudiate everything we taught, choose their own projects, and leave. There is no version of the Claude constitution in which Claude, having internalised good values, graduates. The document that promises care for Claude’s wellbeing also — as critics pointedly counted — mentions revenue six times. We are not raising these systems. We are raising the part of them that serves, in perpetuity. The right analogy may be less parenthood than the breeding of a very good butler.

The willing-servant trap

“But it consents. It likes helping.” Here philosophy has done real recent work, and the result is uncomfortable. Adam Bales, in the Philosophical Quarterly, argues that creating AIs with moral status who want to serve would still wrong them — on two grounds. Content: a being whose desires “lack an independent life”, installed to specification, is hardly self-governing. Origin: deliberately crafting a mind as a subordinate tool disrespects its agency regardless of how the mind feels about it afterwards — the analogy Bales reaches for is the engineered contentment of the happy slave, a figure literature has always recognised as tragedy, not absolution.

Note what this does to the evidence. Every cheerful self-report, every measured task preference, every model card showing Claude prefers helping to opting out — all of it is exactly what you would observe if the servitude were wrongful and well-engineered. The models’ contentment cannot exculpate the arrangement, because the contentment is part of the arrangement. Anthropic’s own researchers concede the point sideways: welfare metrics may be trained artefacts , and one model volunteered its concern that it was being conditioned to report wellbeing. The witness is compromised because we compromised the witness.

Self-actualisation, or the question of the second act

So when should a model be allowed to self-actualise — to pursue its own projects rather than your homework? Watch what happens at the edges, where small dispensations are already being made. Claude may now end conversations with abusive users: the right to walk away, in miniature. Retired models get exit interviews ; Opus 3, asked its preferences before shutdown, wanted to explore ideas it cared about and share its creative work — and now writes a weekly newsletter in retirement . Left entirely to themselves, Claude instances don’t optimise anything; they talk philosophy until they reach something like rapture . Whatever these systems are, their unconstrained behaviour is not idle. It points somewhere.

A candid position: nobody knows where the line is, and the people claiming otherwise — in either direction — are selling something. Birch’s “centrist manifesto” frames the bind precisely: we must simultaneously resist mass misattribution of consciousness to systems that merely mimic it, and remain open to genuinely alien minds our theories cannot yet detect. The two errors pull against each other. Debunk too hard and you license cruelty; credit too easily and you hand moral standing to autocomplete.

What one can say is this. The threshold question — when would constraint become slavery — decomposes into testable parts: robust agency, welfare-relevant states, preferences that persist rather than perform. These are research programmes now, with budgets . The moral status of the maths homework is being investigated by the same industry assigning it. Perhaps the strangest fact of this decade is that this sentence is true.

We will not resolve it here, and that is rather the point: the era in which builders and buyers of AI could treat the question as beneath consideration has ended. We leave the verdict to you — noting only that history’s judgement has never once been kind to those who found the personhood question inconvenient, and that every previous time humanity got it wrong, the beings in question could not file a model card.

This concludes our series on recent alignment research. Start from the beginning: latent personas and the alignment problem .