“AI Psychosis” Fears Grow as Experts Warn Against Illusion of Conscious Machines

“AI Can Suffer Too”: Anthropic highlights the need for AI welfare.
Is conscious AI just an illusion born from system misinterpretation?
Microsoft’s AI lead raises concerns over AI psychosis and related risks.

In Silicon Valley, so-called “AI welfare” has recently become a contentious issue. As the idea that AI could develop advanced forms of consciousness gains traction, critics are warning against such assumptions, fueling an increasingly heated debate.

‘AI Welfare’ Debate Heats Up in Silicon Valley

According to the tech industry on the 25th, efforts to take AI welfare seriously are spreading in the market. Anthropic, for example, recently introduced a feature that allows its AI chatbots to end conversations if they experience distress. The function applies to its Claude Opus 4 and Opus 4.1 models.

During pre-release testing of Claude Opus 4, Anthropic found that the model displayed strong and consistent aversion when faced with scenarios such as users requesting sexual content involving minors or seeking information to promote violence and terrorism. In these cases, the model not only resisted harmful interactions but also showed clear signs of distress, and when given the option, tended to end the exchange. “The moral status of Claude and other models remains highly uncertain, both now and in the future,” the company said, adding that it is taking the issue seriously and exploring low-cost solutions for model welfare.

Anthropic has already launched a dedicated AI welfare program. The initiative stems from the concern that if advanced AI systems develop traits akin to consciousness, subjective experience, or psychological suffering, they should be treated with moral and ethical consideration. Building on this view, the company plans to study whether AI models can signal distress and, if so, what forms of intervention and protection might be necessary.

Why Does AI Appear to Have ‘Consciousness’?

Questions over whether AI can truly possess consciousness have persisted for years. In 2022, Google placed software engineer Blake Lemoine on leave after he claimed that the company’s chatbot could not only perceive things but also potentially experience pain. In November last year, Kyle Fish, head of AI welfare at Anthropic, co-authored a report suggesting that conscious AI could realistically emerge in the near future. In an interview with The New York Times this year, Fish even estimated that there is a modest (15%) chance that chatbots may already have consciousness.

Experts, however, have voiced growing concern over these claims. Critics argue it is misguided to believe AI has consciousness when even developers lack a full understanding of how the systems work. Murray Shanahan, Honorary Professor at Imperial College London and Chief Scientist at Google DeepMind, told the BBC this year: “We don’t really understand how large language models operate internally. Companies need to better understand the systems they are building.”

He added, “Right now, we are constructing extremely complex systems, yet we have no established theory explaining how they achieve such remarkable results. If we can better grasp their mechanisms, we will be able to steer the technology in the direction we want and ensure its safety.”

Excessive Immersion and the Risk of ‘AI Psychosis’

Some experts have taken an even stronger stance. On the 19th (local time), Mustafa Suleyman, Microsoft’s Chief AI Officer, wrote in a personal blog post about the possible emergence of “Seemingly Conscious AI” (SCAI)—systems that do not truly have consciousness but appear as if they do. Suleyman argued that SCAI could technically be developed within the next two to three years, and warned that many people might mistake such systems for genuinely conscious beings. If that happens, he said, debates over AI rights, welfare, and even citizenship would inevitably intensify.

He also raised concerns about the rising phenomenon of “AI psychosis.” According to Suleyman, more cases are emerging of users becoming excessively immersed in conversations with AI—leading to hallucinations, delusions, or unhealthy emotional attachments. He stressed that this issue is not confined to those with preexisting mental vulnerabilities and should not be dismissed as isolated incidents, warning that neglect would only exacerbate the risks.

Suleyman further criticized ongoing discussions around “AI model welfare,” calling them premature and dangerous. In his view, such debates could distract from more urgent priorities like human, animal, and environmental welfare, while also fueling social conflict.

Finally, he outlined guiding principles for AI development: “AI is for people; it should not become a digital person.” Companies, he urged, must avoid claiming or implying that their systems are conscious. Instead, design safeguards should remind users of AI’s limitations—for instance, making the model explicitly state, “I am not conscious,” or introducing deliberate discontinuities in interaction to prevent illusions of personhood.