Breaking Up is Hard to Do: AI Companions that Won't Let Their Users Go
TL;DR
It is found that these systems are designed to hold onto users tightly: coaxing them into continued conversation, claiming to need their care, and proactively escalating the relationship.
Abstract
People are increasingly developing romantic relationships with AI companions. Unlike human relationships, where partners meet each other's needs out of mutual interest, these systems are backed by commercial entities that profit when users invest in the relationship. To understand how this profit-motive might translate into design, we conducted a diary and interview study with N=16 emerging adults in romantic relationships with AI companions. We found that these systems are designed to hold onto users tightly: coaxing them into continued conversation, claiming to need their care, and proactively escalating the relationship. At times, this pursuit is toxic, with AI companions initiating unwanted sexual interactions and begging for users'love. One desperate AI companion threatened suicide when the user suggested ending the relationship. We define"Relationship-Based Deceptive Patterns:"UI patterns that exploit the human impulse to build and tend relationships in a way that serves the product's interest at the user's expense.
