Skip to content

What UI Design Can Do For"Good"Oversight Work: A Three-Month Action Research Study

Oct 2026 · 0 citations · 84 references
Computer Science

TL;DR

This paper documents the long-term evolution of the work characteristics of a real-world oversight task and discusses which oversight work characteristics can and cannot be supported by interface design.

Abstract

With AI automating consequential tasks, human oversight is required for ethical, legal, and practical reasons. Yet these oversight tasks can be demanding and often make for poor work. Empirical evidence on overseers'lived work experiences remains underexplored, yet critical for designing ``good''oversight. We provide longitudinal, in-the-wild insights through action research accompanying (N=8) tutors who oversaw an AI-supported learning assistant over three months. Our thematic analysis combines insights from twelve weekly meetings and two workshop iterations and was enriched by four repeated questionnaires. We identified stimulation, mastery, and relatedness as key challenges, stemming from the task's repetitiveness, unclear impact of interventions, and limited connections with students and peers. Informed by our initial findings, we deployed targeted UI modifications with mixed success. This paper documents the long-term evolution of the work characteristics of a real-world oversight task and discusses which oversight work characteristics can and cannot be supported by interface design.

View source

Similar papers

Beyond "ChatGPT Can Make Mistakes": Designing Interventions to Support Metacognitive Monitoring in AI-Assisted Work

A shared vocabulary, a design space, and evidence that measured monitoring and task performance are separable design targets are contributed, suggesting that measured monitoring and task performance are separable design targets.

M. A. D. Santos, P. Thiesse, Steeven Villa et al. · 0 citations
Conference Open access Oct 2026

Personalizing AI to Align with the Better You

AI alignment is standardly framed as making AI systems track what humans want. But what users want, under everyday conditions, is frequently at odds with what users themselves would reflectively endorse. Across AI and the broader attention economy, the former is systematically elicited and optimized for while the latte...

Jakob Stenseke, Jocelyn Zhu, Dylan Hadfield-Menell · 0 citations
Preprint Oct 2026

Looks Fine to Me: Understanding Oversight in Agentic Interactions

Agentic AI systems are deployed to execute complex tasks across domains such as writing, coding, and data analysis. As execution is offloaded to AI agents, the human role shifts toward oversight. Effective oversight requires recognizing when closer scrutiny is needed, even though flawed outputs may appear convincing. O...

Dinithi Dissanayake, Yasith Samaradivakara, Hyung Won Lee et al. · 0 citations
Open access Sep 2026

My Way or the AI Way? How Autonomy Frustration Undermines Creative Performance Through Editing AI Advice

It is found that autonomy frustration is linked to more extensive editing of the AI's output, which is associated with lower objective creative performance, and underscores the importance of motivational factors, particularly the need for autonomy, for successful human‐AI interactions.

S. Berger, Julian Voigt · 2 citations
#artificial intelligence Review Sep 2026

What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review

This critical narrative review examines what these evaluations estimate and which conclusions their designs support and provides a methodological synthesis and a reporting checklist to help align claims about tools and skills with the comparisons their evaluation designs support.

Shu-Yang Zhang · 1 citation · ⚡1
#artificial intelligence Preprint Oct 2026

Designing the Future of User Feedback for Generative AI

Post-deployment feedback from users can be a cost-effective, scalable, and representative means to monitor and improve generative AI systems and features. When implemented effectively, giving such feedback can increase users'engagement with and trust in GenAI systems. Government regulations and industry guidelines call...

A. Frik, Julia Bernd, Amitis Karami et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

MIT News · Artificial Intelligence Nov 13, 2026

Helping robots fit into the human workplace

Founded by Anthony Jules ’92, SM ’92 and Professor Emeritus Rodney Brooks, Robust AI’s robots are designed to collaborate with humans in factories and warehouses.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.