Audo-Sight: AI-driven Ambient Perception Across Edge-Cloud for Blind and Low Vision Users

Abstract

Despite advances in assistive technologies, Blind and Low-Vision (BLV) individuals continue to face challenges in understanding their surroundings. Delivering concise, useful, and timely scene descriptions for ambient perception remains a long-standing problem in accessibility. Existing solutions often fail to identify user expectations for real-time and accessible responses. Moreover, for a given task, they either rely on cloud offloading, which imposes a significant delay, or edge-based AI, which often sacrifices accuracy. To address this, we present Audo-Sight, an AI-driven assistive system that spans across Edge-Cloud continuum and enables BLV individuals to perceive their surroundings through voice-based conversation. Audo-Sight provides low-latency, accurate, and human-friendly responses through a novel mechanism that seamlessly fuses Edge and Cloud responses. The system also addresses challenges in catering to BLV users through response editing informed by BLV needs. Audo-Sight orchestrates a set of AI models based on user query contextual analysis to infer intent and adjust for a variety of situations. In urgent cases where users require fast responses, Audo-Sight leverages parallel Edge and Cloud pipelines and seamlessly combines responses through its Response Fusion Engine. Systematic evaluation shows that Audo-Sight delivers speech output around 80% faster for urgent tasks and generates complete responses approximately 50% faster across all tasks compared to a commercial cloud-based solution---highlighting the need for customized AI-based solutions. Human evaluation of Audo-Sight shows that it is the preferred choice over GPT-5 for 62% of BLV participants with another 23% stating both perform comparably. Speed and interruption evaluations demonstrate that in most situations, the system can seamlessly respond at a rapid pace to keep up with BLV expectations.