Beyond Captions: Shaping Imagery Models for Blind or Visually Impaired People During Interacting with AI-powered Conversational Visual Assistant
TL;DR
This work examines how mental models of images are built and pruned through dialogue between BVI users and an AI-powered conversational visual assistant, and derives prompt guidelines for layered descriptions, subject-aware defaults and adaptive follow-up suggestions.
Abstract
Digital image-centric platforms create new barriers for blind and visually impaired (BVI) people. We examine how mental models of images are built and pruned through dialogue between BVI users and an AI-powered conversational visual assistant. In a study with 10 BVI participants, we collected 30 voice-based conversations about images showing widely salient objects, allowing unrestricted follow-up questions from users. Exploratory results: AI misses critical details in initial captions; Step-wise analysis shows a common trajectory: users start with a compact scene overview, then request category-specific details (e.g. social cues for people, environmental and functional context for animals and vehicles). Comparing AI-assistant outputs with participants’ later verbal reconstructions reveals both accumulation and pruning: in total, correctly reflected claims increase from 56 to 191 while omitted claims rise from 7 to 121, and mental models similarity scores leveled around 61–63% after Step 4. We derive prompt guidelines for layered descriptions, subject-aware defaults and adaptive follow-up suggestions.