NOVA-HOI: Notions From Oracles for Verb-Agnostic HOI Detection

Abstract

Human-Object Interaction (HOI) detection has long been constrained to fixed sets of predefined interaction categories, limiting generalization to novel verbs and unseen verb-object combinations. While recent work addresses this through zero-shot protocols that partition HICO-DET into seen and unseen categories, this approach is self-defeating: withholding interaction categories from an already data-scarce task reduces training diversity, which is precisely what generic verb understanding requires. This limitation is particularly acute for lightweight models trained exclusively on supervised HOI data, which are structurally disadvantaged relative to VLM (Vision Language Model)-based architectures that implicitly benefit from web-scale pretraining, making zero-shot splits an uneven basis for comparison. Moreover, VLM-based architectures are generally more resource-intensive. Very few methods address genuine open-vocabulary scenarios that evaluate models across different datasets with potentially different verb semantics without sacrificing training data. However, MLLMs (Multi-modal Large Language Models) offer large-vocabulary capabilities but they struggle with accurate interaction localization, are prone to relation hallucination, and are computationally prohibitive for practical applications. In this paper, we propose NOVA-HOI, a teacher-student framework that leverages MLLMs’ description capabilities to pseudo-annotate new data for HOI detection. Crucially, we leverage these models exclusively for offline HOI annotation generation, while maintaining a lightweight detector suitable for practical deployment. Our teacher-student framework integrates these pseudo-annotations using a novel semantic filter, enabling training of an open-verb HOI detector. To rigorously evaluate unseen verb capabilities, we introduce HICOCO, a new evaluation dataset featuring 131 novel verbs and 310 novel HOI classes that systematically tests generalization to unseen interactions beyond HICO-DET. Direct cross-dataset transfer validates our method’s open-vocabulary HOI detection capability, achieving state-of-the-art results with + 4 percentage points mAP role (scenario 2) on V-COCO and + 7 mAP on HICOCO, while remaining competitive with closed-set expert models at a fraction of their computational cost. HICOCO annotations are publicly available at https://github.com/ CEA-LIST/HICOCO