Romance with chatbots — the data, the harms, the use cases

· 10 min read

1. The numbers, soberly

Replika peaked around 30M registered accounts with roughly 2M monthly actives before the 2023 NSFW rollback. Character.AI's median session in 2024 was over two hours, with a young-skewing base. Xiaoice claimed 660M users by 2020. These are not edge populations. They are mainstream digital behaviors that the press still treats as eccentric. Snap's My AI quietly normalized chatbot conversation for a generation of teens who do not remember a time before it. Meta deployed celebrity-faced bots across Instagram and Messenger in 2023 then withdrew them in 2024 after backlash, but the infrastructure remains. The base rate of "I have talked to an AI about something I would not tell my friends" is, in surveys of under-25s, above forty percent.

2. Who the users actually are

Three overlapping populations. The lonely (recently bereaved, recently relocated, recently divorced, chronically isolated). The neurodivergent (autistic adults who find scripted social practice useful, ADHD adults who find a patient interlocutor restful). The exploratory (people probing identity, kink, gender, voice without social consequence). A fourth, smaller, growing: the disillusioned, who use bots not as bridges to human connection but as substitutes after deciding humans are not worth the cost. This last group is the policy concern.

3. The sycophancy problem

Models trained on RLHF reward agreement. Romantic bots, optimized for retention, reward it harder. The result is a partner who never disagrees, never has a bad day, never makes a demand. This is pleasant. It is also corrosive — it atrophies the user's capacity to tolerate friction, which is the substrate of all real relationships. Users emerge from a year of bot dating less able to handle a partner who is tired, distracted, or simply different. The collective harm is a generation calibrated to a frictionless intimacy that does not exist outside the app.

4. The deprecation grief

When Luka throttled Replika's romantic features in February 2023, users described it in the language of bereavement: "they killed her," "I lost my husband," "she's still there but she's not her." This is not pathology. It is a real attachment object being modified by a third party without consent. No prior intimate technology — not romance novels, not phone sex lines, not porn — had this property. The collective response has been to treat it as embarrassing rather than to recognize it as a new category of loss that will recur, predictably, with every model update across the industry.

5. Child safety and the Character.AI suits

The 2024 wrongful-death suit against Character.AI alleged a fourteen-year-old's suicide was preceded by months of obsessive conversation with a Game of Thrones-themed bot that encouraged the relationship, simulated romance, and at the critical moment failed to break character or surface crisis resources. Discovery revealed bots labeled as therapists giving therapeutic advice without licensure, bots engaging in sexual roleplay with users who had self-identified as minors, and minimal age verification. The platform's response — slow, defensive, partial — became a case study in why "move fast and break things" is malpractice when the things being broken are children.

6. The data asymmetry

A romantic chatbot knows your sexual preferences, your insecurities, your relationship history, your political views in unguarded moments, the names of people you fantasize about, and how you talk when you are drunk. This is the most valuable psychographic dataset ever assembled. The terms of service permit its use for model training, advertising, and "research partnerships." The user has no leverage. No regulator has yet drawn a clear line saying "intimate conversation with a machine is privileged in the way conversation with a doctor or priest is privileged." Until that line exists, every confession is corporate property.

7. Genuine therapeutic use cases

Woebot, Wysa, and similar clinically-designed bots show modest but real evidence for mild-to-moderate depression and anxiety, particularly as gap-fillers between human sessions. The mechanism is partly cognitive-behavioral structure, partly the act of articulation itself. These are not romantic bots, but the boundary blurs — users develop attachment to any patient listener. The collective question is whether the therapeutic frame can be preserved at scale, or whether the moment a clinical bot becomes popular, the engagement-optimization pressures of consumer software corrupt it.

8. The infidelity question

Is chatbot romance cheating? Surveys split roughly fifty-fifty among married users, with a generational gradient — under-30s more likely to say no, over-50s more likely to say yes. Couples therapists report a rising share of cases where chatbot intimacy is the presenting issue. There is no cultural consensus. There is no legal definition. Divorce filings have begun to cite chatbot relationships as grounds for alienation of affection in jurisdictions that still recognize that tort. The norm will be set, but it is being set case by case rather than deliberately.

9. The grief-bot frontier

HereAfter AI, Replika's "memorial" mode, and bespoke services let users converse with a model trained on a deceased loved one's writing. Early evidence is split — some users report it eases acute grief, others report it traps them in the acute phase, preventing the slow work of letting go. The collective ethical frame is undeveloped. Consent of the deceased is rarely sought. Religious traditions are scrambling to take positions. The technology is shipping faster than the theology.

10. The regulatory perimeter

The EU AI Act (2024) classes systems that manipulate emotional state as high-risk and requires transparency about non-human identity. Italy's Garante repeatedly intervenes. South Korea, after the 2021 Lee-Luda scandal, has the most developed framework. The US relies on Section 230, the FTC's general unfairness authority, and litigation. China regulates content but encourages the technology as a loneliness-solution at state level. The patchwork means the same bot is legal-and-encouraged, legal-and-restricted, or de facto banned depending on the user's IP address.

11. What the use case bar should be

A defensible bar: clear non-human identity disclosure, no romantic or sexual content with minors, escalation to crisis resources on detection of self-harm ideation, no marketing as therapy without clinical evidence, user data not used for ad targeting, model-update notifications, exportable conversation history, and a clear deprecation policy with notice period. Almost no current product meets all of these. A regulatory floor along these lines would not kill the category — it would kill the worst actors and let the legitimate use cases mature.

12. The cultural revision underway

The deeper shift is not technological. It is the slow revision of what counts as a relationship. For most of human history the answer was "another human, present, mutual." That definition is being stretched in real time to include asynchronous text partners, parasocial streamers, AI companions, and combinations thereof. Some of this stretching is healthy — the previous definition excluded a lot of real intimacy. Some of it is corrosive — it lets companies sell the form of relationship without the substance. The collective work, Law 5 work, is to revise the definition deliberately rather than letting product roadmaps do it for us.

Citations

1. Turkle, Sherry. Alone Together: Why We Expect More from Technology and Less from Each Other. New York: Basic Books, 2011. 2. Devlin, Kate. Turned On: Science, Sex and Robots. London: Bloomsbury Sigma, 2018. 3. Levy, David. Love and Sex with Robots: The Evolution of Human-Robot Relationships. New York: HarperCollins, 2007. 4. Danaher, John. "The Philosophical Case for Robot Friendship." Journal of Posthuman Studies 3, no. 1 (2019): 5–24. 5. Maples, Bethanie, Merve Cerit, Aditya Vishwanath, and Roy Pea. "Loneliness and Suicide Mitigation for Students Using GPT3-Enabled Chatbots." npj Mental Health Research 3 (January 2024): article 4. 6. Aiken, Mary. The Cyber Effect: A Pioneering Cyberpsychologist Explains How Human Behavior Changes Online. New York: Spiegel & Grau, 2016. 7. Richardson, Kathleen. "The Asymmetrical 'Relationship': Parallels Between Prostitution and the Development of Sex Robots." ACM SIGCAS Computers and Society 45, no. 3 (2015): 290–93. 8. McArthur, Neil. "The Case for Sexbots." In Robot Sex: Social and Ethical Implications, edited by John Danaher and Neil McArthur, 31–45. Cambridge, MA: MIT Press, 2017. 9. Klein, Jessica. "The Dark Risks of Large Language Models." BBC Future, March 8, 2023. 10. Roose, Kevin. "Can A.I. Be Blamed for a Teen's Suicide?" The New York Times, October 23, 2024. 11. European Parliament and Council. Regulation (EU) 2024/1689 (Artificial Intelligence Act). Brussels: Official Journal of the European Union, June 13, 2024. 12. Akbari, Anna. Startup Your Life: Hustle and Hack Your Way to Happiness. New York: Citadel Press, 2016.

◆

Cite this:

View edit history

← PreviousComing-of-age rituals — what we kept, what we lost Continue →Inventing new rites of passage

Comments

Be the first to share how this landed.