Interactively Teaching Collaborative Physical Tasks to Generative AI with Narrated Demonstrations

Abstract

Enabling users to teach computers how to perform physical tasks is a long-standing goal in HCI. Prior work has explored learning from demonstration, but demonstrations often underspecify intent. Advances in generative AI enable a combination of demonstration and verbal narration to disambiguate intent. However, it remains largely unexplored how to design interfaces that leverage such narrated demonstrations to teach collaborative physical tasks. To probe this design space, we present a prototype system, Apprento, that models physical tasks as programs via generative AI with three integrated interfaces: (1) a teaching interface for providing narrated demonstrations in mixed reality, (2) an inspection interface for visualizing generated programs as decision-flow diagrams and program executions, and (3) a correction interface for iteratively refining generated programs. Across these interfaces, narrated demonstrations serve as a communication modality between users and the system. In a study with 20 participants teaching soccer skills, we identify differing user expectations around authoring and teaching collaborative tasks, as well as key design tensions between task complexity and program interpretability, and between program expressivity and accuracy. We discuss implications for designing future systems that support symbolic modeling of collaborative physical tasks through narrated demonstrations.