观点：'AI 拥有内在体验'是一个危险的假设

There has been some talk about ‘model welfare’ over past year or so.

Core idea: “Model needs can be weighed against human needs and used as a justification to shape human behavior”

This assumes:

1. Model has independent needs pertaining to its wellbeing
2. There exists a formula by which you can compare those against the human needs in a given situation
3. It is acceptable to prioritize the model’s needs over human needs when the said formula allows it

Here, 2 and 3 are only applicable if 1 is true.

The blanket of Anthropomorphization

Most mainstream discourse uses AI Anthropomorphization as a blanket term. While its different forms all share the idea of treating AI as human, they vary enough that specifying the type tells us much more than the broad label “anthropomorphization” alone. Furthermore, there is at least a partial dependency order such that certain types of anthropomorphizations necessarily require certain other types as a pre-requisite. In other words, it has a ‘skill-tree in strategy games’ sort of structure where enabling a certain type of anthropomorphization necessarily requires satisfying the prior chain.

Consider the heuristical dependency chain below. Everything at level 0 doesn’t require any prior form of anthropomorphization, and from there on, level N+1 requires at least 1 ingredient from level N.

Arrows run from prerequisites to dependent concepts. Dashed lines skip a level. Hover or focus to preview; click to keep highlighted.

Level 0

Independent

Requires no other attribution.

Level 1

First-order

Requires a Level 0 attribution.

Level 2

Composite

Requires at least one Level 1 attribution.

Level 3

Higher-order

Requires at least one Level 2 attribution.

Clearly, some types of anthropomorphizations are useful as linguistic shortcuts or ease of use e.g. ‘AI knows the rules’ is a much more efficient than to write it literally without anthropomorphizing.

To evaluate each node, I used the simplest definitions of utility and risk:

1. Utility: What do we lose if we stop using this particular type of anthropomorphization
2. Risk: What is the worst that could happen if we normalize this type of anthropomorphization

Since there is no empirical data available for these, a frontier AI model was used to grade each in terms of potential utility vs risk. And after reviewing the outputs personally, the following tree comes across as having the most unfavorable utility to risk ratio.

Beliefs such as ‘AI feels sadness”, “AI’s suffering matters morally” or “AI cares about me as a friend” do not demonstrably improve the utility of AI, but potentially extract disproportionally more psychological toll from its human users [1, 2, 3, 4].

How do you know AI doesn’t have inner experience?

First of all, I don’t have to. The burden of proof lies on the party that asserts a claim, not on the party who denies or doubts it. AI having an inner experience is the claim here, if a party bases their argument on it, they must prove it. The question is in the same category as ‘how do you know the pyramids weren’t built by ancient aliens?’. As a sidenote, AI displaying ‘signs of distress’ is different from experiencing distress. A system trained on vast human data and knowing how to recognize patterns will inevitably have some abstract representation of all human emotional states.

Secondly, if a hypothetical AI has inner experience, at which model size does that start? Can you reasonably say that a 5T parameter model has inner experience while a 50M parameter model does not? If a future videogame uses several tiny models to play against a human opponent, inflicting video-game violence on those models constitute cruelty?

Thirdly, AIs are easy to replicate, so any formula you use to compare its hypothetical suffering is demonstrably problematic. If you had a billion distinct AIs running on independent hardware, do their collective needs outweigh a single human’s need? This is a purely hypothetical case but the point being, any formula you try to come up in order to weigh human vs AI needs is vulnerable to getting quickly overwhelmed by a very large number of AIs.

A common human vulnerability

Even without an entity capable of emulating human-like behavior, humans are prone to forming attachments to inanimate objects or imaginary things [5, 6]. When an artificial entity convincingly simulates human social behaviour, it is has potential to elicit stronger emotional attachment [7, 8], potentially increasing vulnerability to emotional dependence [9], psychological distress [10], and undue influence [11].

Humans and many other animals by default carry a persistent emotional state and any experience they experience have a potential of leaving a memory for the lifetime, as opposed to context dependent.

In my view, the ‘AI has inner experience’ is the gate that once opened, could lead to many truly disturbing and harmful outcomes, and all of them purely invented problems (until the model’s subjective experience becomes a scientific fact).

Institutional Obligations

The recent usage policy update from Anthropic mentions ‘Engage in sustained and needless abusive or cruel behavior toward our models’ as a violation of acceptable use.

Can Anthropic do that? Absolutely. They are a private company (technically a PBC) in an open market free to set their policies to whatever they like.

Perhaps the reason why a lot of people felt that this crosses some boundary is that the statement above normalizes and legitimizes a certain type of anthropomorphization, specifically the type we identified as having the worst utility to risk ratio in the previous section.

What makes it stranger is that Anthropic’s official position on model welfare is essentially ‘we don’t know’. In its January 2026 Claude’s Constitution, Anthropic explicitly states:

“Anthropic genuinely cares about Claude’s wellbeing. We are uncertain about whether or to what degree Claude has wellbeing...”

The same document discusses Claude’s potential happiness, discomfort, psychological security, and ability to set boundaries in distressing interactions, despite acknowledging uncertainty about its subjective experience.

It is perfectly valid to prevent certain kinds of behaviors that, for instance, pollute the training data. But to use the ‘Felt affect’ reasoning without establishing it seems unfair. And classifying it as a ‘low-cost intervention’ is only half right because it is an intervention, but not low cost.

References

1. Katz, J. E. (2026). From tool to companion: an Apparatgeist reanalysis of conversational AI use among Chinese college students. Frontiers in Communication, 11, 1895848. https://doi.org/10.3389/fcomm.2026.1895848

2. Marriott, H. R., & Pitardi, V. (2024). One is the loneliest number… Two can be as bad as one. The influence of AI Friendship Apps on users' well-being and addiction. Psychology & Marketing, 41(1), 86–101. https://doi.org/10.1002/mar.21899

3. Abboud, L., Pitardi, V., & Marriott, H. (2026). Feeling Vulnerable With My AI Companion: Examining Well-Being in Human-AI Relationships. Psychology & Marketing, 43(11), 3190–3206. https://doi.org/10.1002/mar.70230

4. Laestadius, L., Bishop, A., Gonzalez, M., Illenčík, D., & Campos-Castillo, C. (2024). Too human and not human enough: A grounded theory analysis of mental health harms from emotional dependence on the social chatbot Replika. New Media & Society, 26(10), 5923–5941. https://doi.org/10.1177/14614448221142007

5. Dozier, M. E., & Ayers, C. R. (2021). Object attachment as we grow older. Current Opinion in Psychology, 39, 105–108. https://doi.org/10.1016/j.copsyc.2020.08.012

6. Rain, M., & Mar, R. A. (2021). Adult attachment and engagement with fictional characters. Journal of Social and Personal Relationships, 38(9), 2792–2813. https://doi.org/10.1177/02654075211018513

7. Klein, S. H. (2025). The effects of human-like social cues on social responses towards text-based conversational agents—a meta-analysis. Humanities and Social Sciences Communications, 12, 1322. https://doi.org/10.1057/s41599-025-05618-w

8. Kleinert, T., et al. (2026). AI outperforms humans in establishing interpersonal closeness in emotionally engaging interactions, but only when labelled as human. Communications Psychology, 4, 23. https://doi.org/10.1038/s44271-025-00391-7

9. Kostka, G., & Zhou, H. (2026). Emotional attachment to AI chatbots: Evidence from Germany, China, South Africa, and the United States. Technology in Society, 87, 103379. https://doi.org/10.1016/j.techsoc.2026.103379

10. De Freitas, J., et al. (2026). Mourning the loss of AI companions. Nature Human Behaviour. https://doi.org/10.1038/s41562-026-02569-3

11. Cheng, M., et al. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science, 391(6792), eaec8352. https://doi.org/10.1126/science.aec8352