Human-Centered Metrics for Evaluating Usability in Artificial Intelligence Generated Human Object Interactions

Abstract

Usability is key to assessing whether interactions reflect real-world engagement with designed artifacts. However, evaluating usability requires design and prototyping, which incur substantial time and financial costs. With recent advances in multimodal generative AI (GenAI), it has become possible to predict, to some extent, the usability that a given design will afford, offering a potential solution to this problem. However, existing approaches focus primarily on visual fidelity and lack systematic evaluation of interaction interpretability from a human-centered perspective. To address this gap, the present study introduces the concept of interaction GenAI Affordance Quotient (AI-AQ), a human-centered framework for evaluating human-object interaction(HOI) quality. AI-AQ decomposes interactions into three components: subject consistency, affordance quality, and common sense alignment, enabling structured assessment. We compare GenAI-generated HOIs with real HOIs across varying object familiarity through a two-stage user study (N=80 pilot; N=800 main), using object images and textual prompts in GPT-4o. Results reveal a fundamental boundary condition: when evaluators possess semantic but not procedural knowledge of an object, neither structured metrics nor intuitive judgments can reliably differentiate AI-generated from real interactions — a systematic evaluation collapse that is absent when knowledge is complete or entirely absent. Beyond this, GenAI interactions are consistently inferior to real interactions across all dimensions. These findings suggest that multimodal generative models capture high-level visual and semantic patterns but remain limited in representing fine-grained interaction structure. This work contributes to human-aligned multimodal interaction and supports AI–human co-design by bridging visual, behavioral, and cognitive aspects of interaction.