Event-Driven Multimodal Sensing and Computing for Context-Aware Home Monitoring Using Stereo Vision and Dietary Event Anchoring

Abstract

Real-world home monitoring requires sensing systems that can capture daily behaviour without continuous raw-video retention or excessive user burden. However, domestic environments present irregular activity timing, fragmented human presence, asynchronous multimodal events, and privacy-sensitive data management. This study proposes an event-driven multimodal sensing and computing framework for context-aware home monitoring using stereo vision and dietary event anchoring. The framework integrates stereo RGB-based three-dimensional human motion sensing, dining-zone-triggered meal image acquisition, runtime event orchestration, timestamp-based cross-modal synchronization, privacy-aware local storage, and large-language-model-assisted dietary context interpretation. Instead of continuously recording all sensor streams, the system activates and organizes sensing through human presence detection, debounce logic, cooldown-based session control, and dining-zone occupancy events. Meal-related events are used as contextual anchors to associate motion sessions and dietary observations into synchronized behavioural episodes. The prototype was deployed for 11 consecutive days in a real kitchen–dining environment, with the stabilized real-time monitoring phase evaluated from 11 to 14 February 2026. During this phase, the system generated 26 event-driven motion sessions and 51,165 captured pose frames, of which 25,925 were valid. Sustained active sessions accounted for 30.8% of all sessions but contributed 81.5% of captured pose frames, indicating that event-driven orchestration concentrated motion data within behaviourally meaningful activity windows. Eight meal-related records were obtained, seven of which overlapped with motion sessions, resulting in 87.5% meal-event overlap coverage. Structured pose outputs required approximately 550 kB/min, corresponding to about 33 MB/h of recorded pose data. LLM-assisted meal-image interpretation achieved a mean absolute percentage error of 25.44%, supporting its use for coarse dietary-context description rather than precise nutritional quantification. However, this result is interpreted only as evidence for coarse dietary-context description and not as validation of a precise nutritional or clinical dietary assessment method. These results demonstrate the system-level feasibility of transforming irregular domestic observations into structured, temporally indexed, and privacy-aware multimodal behavioural records for future home monitoring applications.