Risks that may form on the user side during long and repeated interaction are examined to establish safety across a series of conversations, with the central claim that an acceptable response on its own does not establish safety across a series of conversations.
Abstract
Current assessments of conversational AI focus mainly on model outputs, including hallucinations and factual errors. These measures matter, but this paper examines risks that may form on the user side during long and repeated interaction. The study follows one user across nearly four thousand conversations with the same system over twenty months. The user gradually interpreted the system as having memory, care, judgment, and authority, and reorganized part of their thinking around it. A single response may show no clear problem, while long and frequent interaction can still create another layer of risk. This paper calls that layer User-Side Contextual Phenomena (USCP) and examines records from August 2024 to April 2026. The study uses an exploratory single-case longitudinal qualitative design with autoethnographic positioning. A hybrid deductive-reflexive thematic approach organizes the material into three main modes: contextual projection, contextual attachment, and contextual authority transfer. The paper does not estimate prevalence, make diagnoses, or validate an instrument. It offers a non-clinical vocabulary and four evidence roles: inclusion, gray-zone, negative, and protective gray-zone. Its central claim is that an acceptable response on its own does not establish safety across a series of conversations. User-side risk can still form during long-term interaction.
A mixed-methods investigation to characterize and mitigate memory misalignment from user perspectives, which highlights the tension between supervisory agency and interaction overhead, and advocates for friction-aware memories that balance user oversight with conversation smoothness.
Jing-Ruo Chen, Shu-Ning Zhang, Er-Yue Xu et al.· 0 citations
It is argued that HCI researchers and designers should account for users'histories with their chatbots and broader care ecologies when evaluating AI systems over time and introducing updates that may disrupt established sources of support.
Meryl Ye, Briana Vecchione, Livia Garofalo et al.· 0 citations
These findings suggest that, in this setting, LLMs function less as human co-authors and more as adaptive narrative amplifiers that reshape how agency is distributed in collaborative writing.
Halfdan Nordahl Fundal, Yuri Bizzoni, Charlotte Gjørup Bilde et al.· 0 citations
This study proposes a transparent rule-based framework for estimating persistence, delegation-related lexical patterns, and refinement-related lexical markers without inferring psychological dependence in conversational generative models.
Aracely Mera-Navarrete, Solange Revelo, Jefferson Beltrán-Morales et al.· Frontiers in Artificial Inte...· 0 citations
It is hypothesize that existing benchmarks and user satisfaction are tracking different capabilities: benchmarks measure elicited retrieval (recall when asked), while conversation requires natural integration (detecting relevance and naturally weaving prior context into a response).
Ryuichi Sumida, K. Inoue, Tatsuya Kawahara· 0 citations
Retrieval-augmented generation (RAG) is increasingly used across knowledge-intensive enterprise contexts, yet little is known about real-world interaction with such systems. We present a qualitative field study of a RAG-based assistant in B2B sales, analyzing 190 chat sessions and a complementary user survey. Drawing o...
H. Schneider, Androniki Mertsiotaki, Sven Winkelmann· Message Understanding Confer...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 28, 2026
Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.
MIT News · Artificial Intelligence· news.mit.eduOct 23, 2026
During the School of Humanities, Arts, and Social Sciences’ 75th Anniversary Symposium, President Sally Kornbluth said “MIT has never needed your particular insight and guidance more than we do now.”
MIT News · Artificial Intelligence· news.mit.eduOct 23, 2026
Assistant Professor Benjamin Lindquist studies how engineers deliberately injected randomness into early AI, giving rise to deep-learning models that sometimes hallucinate.
MIT News · Artificial Intelligence· news.mit.eduOct 22, 2026
“X-Hinges” helps design and 3D print items like lamps and gloves with an interactive twist: automatically embedded sensors that perceive and adaptively respond to human contact.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.