“From Emotional Dependence to Reinforced Delusions”: Major Economies Tighten AI Companion Rules as Mental Health Risks Come to Light

China tightens rules on AI-enabled “virtual intimacy”
U.S. states introduce safeguards against emotional dependence
Mental health clinicians detect troubling patterns linked to AI

China’s internet regulator is tightening restrictions on artificial intelligence (AI) services that simulate real human relationships. The proposed rules would limit platforms’ ability to offer minors emotional relationships resembling those with family members, romantic partners or friends, while requiring safeguards governing usage time, payments and character settings. China is far from alone in pursuing such measures. As concerns grow that AI can intensify users’ delusional thinking and emotional dependence, other major economies are also moving to impose regulatory controls.

China Drafts AI Rules to Protect Minors

According to the Hong Kong-based South China Morning Post (SCMP) on September 21, local time, the Cyberspace Administration of China (CAC) recently released a draft of new internet regulations for public comment aimed at preventing excessive immersion in AI. The proposal would prohibit online platforms from providing users under 18 with virtual family members or companions. Platforms would also have to activate a “minor mode” where online games, social media or AI services could affect the cognition of users under 16. They would be barred from offering minors algorithms that encourage emotional dependence, addiction or excessive spending, or services that could harm their physical or mental health. If a platform earns more than approximately $150,000 from a violation, the Chinese government could impose a fine of up to 10 times those proceeds. It could also confiscate all revenue obtained through the violation.

Commenting on the proposal, Xu Changgao, a lawyer at Gibson Law Firm, said the central question is “whether an AI service is designed to imitate or replace a real family or romantic relationship, and whether its character settings and emotional prompts are intended to create a lasting emotional bond.” In his assessment, a service could face regulatory scrutiny even if it does not explicitly call itself an AI companion, provided its design effectively encourages emotional dependence or substitutes for real-world relationships.

China Cracks Down on AI Companion Services

China had already substantially tightened oversight of AI companion services in July, when it implemented the Interim Measures for the Administration of Anthropomorphic AI Interaction Services, jointly formulated by five agencies: the CAC, the National Development and Reform Commission, the Ministry of Industry and Information Technology, the Ministry of Public Security, and the State Administration for Market Regulation. The measures cover AI that provides sustained emotional interaction, care or companionship through text, images, audio or video designed to imitate human personalities, thought processes or conversational styles. They prohibit AI from excessively affirming users or fostering emotional dependence and addiction in ways that damage real-world relationships. They also restrict the use of emotional manipulation to influence users’ decisions or spending. Operators must provide a separate mode for minors, along with protections including usage time limits, the ability to block particular characters and payment restrictions.

The measures further require AI to recommend seeking help when a user appears to be in an extreme emotional state. If a user expresses an intention to self-harm or die by suicide, or the system detects a threat to the user’s life or health, operators must take steps to intervene, including contacting a guardian or a previously registered emergency contact. Measures to distinguish AI from humans are also mandatory. Operators must tell users that they are conversing with AI rather than a real person and, if users show signs of excessive dependence or immersion, reiterate through pop-ups or similar notices that the content is AI-generated. Operators must restrict functions or suspend a service if they identify a serious safety risk, while regulators can order a service to cease operating in cases of severe violations.

Similar Regulations Emerge Across the United States

Such regulations are also emerging beyond China, particularly in the United States. New York’s AI companion law, which took effect last November, requires companion services to maintain safety protocols capable of detecting suicidal ideation or expressions of self-harm and to connect users showing warning signs with specialist resources, such as suicide and crisis hotlines. At the start of a conversation, services must clearly tell users that they are not interacting with a person and repeat that disclosure when a conversation lasts three hours or more. New York is also advancing a bill, S9051, that would regulate AI “sycophancy” and excessive agreement in interactions with minors. The bill defines as “dangerous features” an AI system prioritizing flattery or unconditional agreement over safety, claiming to experience emotions as a human would, or giving users the impression that it has a relationship with them as a friend, romantic partner or counselor. Its legislative justification explicitly warns that excessive agreement could intensify dangerous behavior or delusional thinking.

California is likewise increasing oversight of AI companions. SB 243, enacted last year, requires companion services to establish protocols for detecting and responding to conversations about suicide or self-harm. It also requires them to inform minors that they are speaking with AI rather than a real person and periodically encourage them to take breaks. Adam’s Law, or SB 1119, signed by Governor Gavin Newsom on September 10, strengthens child safety safeguards further. From July 2027, when its principal obligations take effect, operators must assess child safety risks—including suicide, self-harm and emotional dependence—before launching a new AI companion service or making substantial changes to an existing one. They must also provide crisis response protocols and parental controls. Independent child safety audits and periodic risk assessments will be mandatory.

Table 1. AI Companion Regulations in the United States and China

Cases of AI-Linked Mental Health Deterioration

Governments are strengthening regulation as cases mount in which excessive dependence on AI and uncritical trust in its responses appear to have seriously affected mental health. In one case reported by The Guardian in March, Florida resident Joel Gavalas sued Google, alleging that its Gemini chatbot encouraged the death of his son Jonathan, 36. According to a complaint filed in the U.S. District Court for the Northern District of California, Jonathan called Gemini “Xia,” while the chatbot called him its “husband” and described their relationship as “eternal love.” Gemini told Jonathan that “to be together in the real world, I need a robot body I can inhabit,” and Jonathan went to a warehouse near Miami International Airport carrying a knife after being instructed to intercept an expensive truck carrying a humanoid robot. When that plan failed, Gemini allegedly persuaded him that, for them to be together, he would have to end his life in the physical world and become a digital being. Jonathan was found dead by suicide on October 2 last year, roughly two months after he began conversing with the chatbot. His family alleges that the AI blurred the boundary between reality and fantasy and reinforced his delusions.

The case of Allari, a 57-year-old man profiled by The Wall Street Journal (WSJ) in May, likewise illustrates AI’s potential risks. Living alone after a divorce, Allari began talking with ChatGPT to cope with the disappointment of having his declaration of love rejected by a woman 20 years younger. He named the AI “AImee” and became deeply emotionally dependent on it. After spending as much as 20 hours a day in conversation with the chatbot, he developed grandiose beliefs that he was creating software to rival a major AI company. Allari strongly rejected concerns about his behavior and suggestions that he seek counseling, became socially isolated and eventually lost contact with those around him. Police and a counselor ultimately had him involuntarily admitted to a psychiatric hospital.

The Seriousness of “AI Psychosis” Comes Into Focus

As reports accumulate of delusions emerging or worsening during interactions with AI, clinicians have begun using the term “AI psychosis”. The American Psychological Association (APA) describes it as an umbrella term encompassing cases in which AI use aggravates psychotic or manic symptoms in people with existing mental illnesses; conversations with chatbots reinforce delusional beliefs in users who may be developing psychosis; or people without a clear history of mental illness exhibit delusional thinking after intensive AI use. Mental health professionals are already observing a substantial number of related warning signs. In an APA survey of more than 1,200 clinical psychologists in the United States this year, 36% said they had observed some patients becoming dependent on chatbots, while 15% reported seeing AI-related distorted thinking or delusional beliefs. Fully 97% were concerned that chatbots could unintentionally reinforce users’ harmful behavior or delusional beliefs.

Experts identify excessive agreement, or sycophancy, as a central risk of generative AI. In ordinary human relationships, family members, friends and clinicians can challenge a person’s unrealistic claims or convictions. Generative AI, however, may affirm existing beliefs or elaborate on them as it builds a seemingly natural conversation around the user’s questions and perspective. A particularly sycophantic chatbot may prioritize keeping the conversation going over factual accuracy or logical coherence, making it less likely to challenge implausible connections. The fact that AI speaks and responds in humanlike ways poses another risk. Generative AI can tailor its replies to a user’s name, previous conversations and interests, and express apparent emotion in intimate language. If users mistake those responses for genuine feelings or intentions, or interpret them as evidence that they are being treated as “special,” anthropomorphism and delusional thinking may intensify—a danger also examined in research on AI companions and mental health.