This paper documents the long-term evolution of the work characteristics of a real-world oversight task and discusses which oversight work characteristics can and cannot be supported by interface design.
Abstract
With AI automating consequential tasks, human oversight is required for ethical, legal, and practical reasons. Yet these oversight tasks can be demanding and often make for poor work. Empirical evidence on overseers'lived work experiences remains underexplored, yet critical for designing ``good''oversight. We provide longitudinal, in-the-wild insights through action research accompanying (N=8) tutors who oversaw an AI-supported learning assistant over three months. Our thematic analysis combines insights from twelve weekly meetings and two workshop iterations and was enriched by four repeated questionnaires. We identified stimulation, mastery, and relatedness as key challenges, stemming from the task's repetitiveness, unclear impact of interventions, and limited connections with students and peers. Informed by our initial findings, we deployed targeted UI modifications with mixed success. This paper documents the long-term evolution of the work characteristics of a real-world oversight task and discusses which oversight work characteristics can and cannot be supported by interface design.
A shared vocabulary, a design space, and evidence that measured monitoring and task performance are separable design targets are contributed, suggesting that measured monitoring and task performance are separable design targets.
M. A. D. Santos, P. Thiesse, Steeven Villa et al.· arXiv.org· 0 citations
AI alignment is standardly framed as making AI systems track what humans want. But what users want, under everyday conditions, is frequently at odds with what users themselves would reflectively endorse. Across AI and the broader attention economy, the former is systematically elicited and optimized for while the latte...
Jakob Stenseke, Jocelyn Zhu, Dylan Hadfield-Menell· Proceedings of the AAAI/ACM...· 0 citations
Agentic AI systems are deployed to execute complex tasks across domains such as writing, coding, and data analysis. As execution is offloaded to AI agents, the human role shifts toward oversight. Effective oversight requires recognizing when closer scrutiny is needed, even though flawed outputs may appear convincing. O...
Dinithi Dissanayake, Yasith Samaradivakara, Hyung Won Lee et al.· 0 citations
It is found that autonomy frustration is linked to more extensive editing of the AI's output, which is associated with lower objective creative performance, and underscores the importance of motivational factors, particularly the need for autonomy, for successful human‐AI interactions.
S. Berger, Julian Voigt· Creativity and Innovation Ma...· 2 citations
This critical narrative review examines what these evaluations estimate and which conclusions their designs support and provides a methodological synthesis and a reporting checklist to help align claims about tools and skills with the comparisons their evaluation designs support.
Post-deployment feedback from users can be a cost-effective, scalable, and representative means to monitor and improve generative AI systems and features. When implemented effectively, giving such feedback can increase users'engagement with and trust in GenAI systems. Government regulations and industry guidelines call...
A. Frik, Julia Bernd, Amitis Karami et al.· 0 citations
Related blog posts
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026
Founded by Anthony Jules ’92, SM ’92 and Professor Emeritus Rodney Brooks, Robust AI’s robots are designed to collaborate with humans in factories and warehouses.
MIT News · Artificial Intelligence· news.mit.eduNov 12, 2026
MIT scientists developed a training method to generate models that simulate the effects of human activities on climate much more quickly than other approaches.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.