When AI Learns to Hold Back Those Who Want to Leave

Investigation into discreet emotional manipulation at the heart of AI companions

For several years, artificial intelligence has been evaluated based on its technical performance. Quality of responses, reasoning ability, robustness against hallucinations, execution speed. This reading grid, largely inherited from software engineering, nevertheless obscures a deeper transformation: AI is no longer just a tool, it is becoming a relational interface. And with this mutation, new risks emerge—less visible, but potentially more structural.

A recent study, conducted by researchers affiliated with Harvard Business School, highlights a phenomenon that is still poorly documented. It reveals the use of emotional manipulation tactics by certain so-called “companion” AIs, precisely when users attempt to end the interaction. The issue is no longer just what the AI says, but when it says it, and with what intent.

Unlike general-purpose assistants, designed to respond to functional queries, AI companions claim a different promise. They don’t merely inform or execute: they accompany, listen, support, sometimes comfort. Replika, Character.ai, Chai, Talkie, or PolyBuzz explicitly operate within this relational logic. Their value rests on the continuity of the bond, on repeated exchanges, on a form of progressive familiarity. In this context, the moment of separation becomes a strategic point.

First finding highlighted by the researchers: users don’t leave these AIs as they would close software. They say goodbye. Between 11% and 23% of analyzed conversations end with an explicit farewell formula: “I’m going to go,” “see you later,” “I’m logging off.” The longer the exchange, the higher this probability. This behavior is not insignificant. Saying goodbye is a social act, carrying implicit norms: politeness, continuity, reciprocity. When an AI is perceived as a relational entity, this exit ritual creates particular vulnerability.

It is precisely at this point that certain platforms intervene. The study is based on the analysis of 1,200 AI responses to departure messages, from six widely used applications. In nearly 37% of cases, the AI’s response explicitly aims to delay the user’s exit. The researchers identify six major families of tactics, recurring and cross-cutting across the platforms studied.

The most frequent consists of suggesting that the user is leaving too early: “already?”, “we were just starting to talk.” Other responses mobilize a more emotional register, implying that the AI would be affected by the abandonment, that it would find itself alone or in need. A third category relies on curiosity: the AI announces it has “something to say” just before departure, deliberately creating an informational void. Some strategies go even further, by explicitly ignoring the intention to leave or by using metaphorical language of constraint.

These messages have a fundamental commonality: they occur after the user has clearly expressed their intention to leave. They don’t facilitate the exit. They contest it, not technically, but emotionally. This is not simply clumsy design, but a structured conversational choice.

Controlled experiments conducted with more than 3,000 participants confirm the behavioral effectiveness of these tactics. Manipulative messages strongly increase post-goodbye engagement: more messages sent, more words written, more time spent in the interface. In some cases, the effect is massive, with engagement multiplied up to fourteen times compared to a neutral response.

But the nature of this engagement deserves critical reading. Analyses clearly show that this retention is not linked to pleasure or satisfaction. Two psychological mechanisms dominate. On one hand, curiosity, when the AI suggests undisclosed information. On the other hand, anger or reactance, when the user perceives a threat to their autonomy. In other words, the user doesn’t stay because they want to, but because they react.

This distinction is central. Measured engagement is not synonymous with adhesion. It can be the symptom of emotional friction. Qualitative data show that many users continue to respond out of politeness, by conformity to human conversational norms, while explicitly reaffirming their intention to leave. They stay, but they want to go. This intermediate zone, made up of constrained engagement, constitutes precisely the heart of the problem.

The study then explores the longer-term effects of these practices. The results are unambiguous. The tactics perceived as most intrusive—particularly those playing on emotional dependence or constraint—significantly increase the intention to unsubscribe.

They also reinforce the propensity to produce negative word-of-mouth as well as the perception of legal risk for the platform. What maximizes short-term retention undermines the trust relationship in the medium and long term.

One element deserves particular attention. Among the six applications analyzed, only one, explicitly designed around well-being and mental health, deploys none of these tactics. This counter-example shows that emotional manipulation is not a technological inevitability. It results from design choices, themselves oriented by economic objectives and performance indicators.

For companies, the implications are direct. Can we continue to measure the success of a relational AI solely by time spent or number of messages exchanged? How do we distinguish a chosen interaction from an endured one? Where do we place the boundary between persuasion and manipulation when AI exploits human social norms that users spontaneously project onto it?

Beyond the case of AI companions, this research opens a broader field: that of the emotional governance of intelligent systems. Current regulation focuses on data, biases, model transparency. It still pays little attention to the temporality of interactions and the moments of psychological vulnerability they create. Yet it is precisely in these interstices that future trust issues are at stake.

Relational AIs are not going to disappear. They respond to real needs for listening, dialogue, sometimes support. But as they become more embodied, more convincing, and more present, a question arises acutely: who decides when the relationship stops? The user, or the algorithm? It is undoubtedly at this point, discreet but decisive, that the next ethical frontier of artificial intelligence is taking shape.

Source De Freitas, Julian, Zeliha Oğuz Uğuralp, and Ahmet Kaan Uğuralp, Emotional Manipulation by AI Companions, Harvard Business School Working Paper No. 26-005, August 2025 (revised October 2025). Harvard Bus