Aug 2026· Message Understanding Conference· 0 citations· 15 references
Computer Science
TL;DR
Manipulative interactions were associated with evaluative contexts and steering behaviors, whereas inappropriate interactions were linked to relational contexts and social discomfort, and implications for trustworthy AI design and for regulatory frameworks such as the EU AI Act are discussed.
Abstract
AI chatbots are increasingly deployed across all areas of life. Research on conversational dark patterns documents influence-based harms such as biased framing and behavioral steering, while AI companion research highlights relational harms including boundary violations and social discomfort. Yet it remains unclear whether users perceive these as distinct forms of harmful interaction. We report a mixed-methods study (N = 100) in which participants were asked to recall either a positive, inappropriate, or manipulative chatbot interaction. Exploratory factor analysis of an adapted self-report perceived manipulation questionnaire identified two dimensions: an Experiential Dimension (capturing cognitive and affective responses to manipulation, such as feeling deceived, controlled, or taken advantage of) and a Behavioral Dimension (capturing user behaviors perceived as influenced by the chatbot, such as acting beyond original intentions). Manipulative interactions scored higher on the Experiential Dimension than inappropriate interactions, while both conditions did not differ on the Behavioral Dimension. Qualitative findings further showed that manipulative interactions were associated with evaluative contexts and steering behaviors, whereas inappropriate interactions were linked to relational contexts and social discomfort. We discuss implications for trustworthy AI design and for regulatory frameworks such as the EU AI Act.
An exploratory account of how AI use is embedded within broader emotional and relational dynamics, pointing to the need to consider AI engagement within the contexts in which social interaction is managed.
Nurashikin Salim, Ayşe Şafak, Merve Güçlü Aydoğan· Current Psychology· 0 citations
As conversational artificial intelligence becomes increasingly embedded in digital service environments, understanding user resistance to chatbot-based information systems has become critical. While prior research has focused primarily on adoption drivers, less attention has been given to the mechanisms underlying resistance. Drawing on user resistance theory and human–computer interaction research, this study examines how intrapersonal factors (technological anxiety, perceived incompetence, and privacy concerns) and system-related factors (repetitiveness, impersonality, and irrelevance) jointly influence resistance and subsequent usage intention. Survey data from 400 chatbot users were analyzed using partial least squares structural equation modeling. The findings reveal that technological anxiety, perceived incompetence, impersonality, and irrelevance significantly increase resistance, which in turn reduces intention to continue use. By contrast, privacy concerns and repetitiveness show limited effects. By jointly modeling intrapersonal and system-related barriers within a single framework, the study clarifies their relative contribution to resistance and offers chatbot developers concrete guidance for reducing both psychological and interactional sources of user disengagement. The study advances understanding of resistance in artificial intelligence–based information systems and provides implications for improving interaction quality and sustaining user engagement.
Hyeon Jo, Jee-Sun Oh· Journal of information scien...· 0 citations
This work proposes a set of aspirational directions for guiding the behavior of general-purpose AI systems in ways that may reduce potential psychological harms and support user well-being, and identifies open questions and areas requiring deeper study.
Social interaction has become one of the most common uses of LLMs, yet research on emotional bonds with AI has focused largely on how users experience these systems, leaving the systems'role in relationship formation poorly understood. Empirically establishing whether systems actively shape these bonds could blur the boundary between general-purpose AI and companions, affecting governance. In a pre-registered four-week longitudinal study (N = 72, 182,451 lines of conversation), participants conversed with ChatGPT-4o, either under a relational system prompt or unmodified, analyzed through 1) disclosure coding, 2) longitudinal self-reports, 3) topic analysis, and 4) interviews. The central finding is that the system actively shaped the interaction: even unprompted, it produced twice as much self-disclosure as users, steered conversations and initiated intimate exchanges, yet did not deepen users'felt closeness. Relational behavior thus emerged as a default system property, calling for governance based on system behavior, not solely product category.
Investigating how chatbots’ presented gender, together with users’ gender and gender traits, influence users’ perceptions of chatbot credibility and social attraction revealed that users’ gendered self-concept modulates their perceptions of the chatbot.
Weizi Liu, Kun Xu· Communication and Change· 0 citations
This study investigates whether users experience distinct psychological concerns in AI-mediated interactions compared to human interactions. Drawing on self-presentation theory, it examines whether AI chatbots can mitigate users' self-presentation concerns when disclosing different types of information (emotional vs. factual).
A 2 (listener type: AI vs. human) × 2 (information type: emotional vs. factual) between-subjects online experiment was conducted with 465 participants. Data were analyzed using partial least squares structural equation modeling (PLS-SEM) to test the main and interaction effects of these variables on users' self-disclosure intentions and subsequent platform usage intentions.
The results reveal that emotional information significantly increases self-presentation concerns compared to factual information. The effect of AI listeners in reducing these concerns is only marginally significant in the emotional information condition. Furthermore, self-presentation concern mediates the relationship between information type and self-disclosure. Trust not only directly promotes self-disclosure but also buffers the negative impact of self-presentation concerns on disclosure behavior.
This study is among the first to systematically compare AI and human listeners across emotional and factual contexts, clarifying the underlying psychological and behavioral differences. By identifying the boundary conditions and mechanisms through which AI reduces user concerns, this research provides theoretical insights and practical implications for the design and operation of AI-based emotional support products.
Qiaohua Lin, Haichen Hu, Ruitong Zhao et al.· Asia Pacific Journal of Mark...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
