HybridLVLM: Semantic Core-Frame Selection and Two-Stage Chain-of-Thought Reasoning for Continuous Driver Action Recognition

Abstract

Driver action recognition (DAR) is a core capability of advanced driver assistance systems, however, it remains difficult because distracted driver actions involve small interacting objects, subtle hand movements, and severe self-occlusion inside a confined cabin. Most existing methods are trained on pre-segmented video clips with predefined action labels requiring substantial annotation effort and additional labeled data when extending to new action classes, which limits their scalability to real-world applications. Moreover, they cannot localize actions in an untrimmed video stream. This paper presents HybridLVLM, a training-free framework for continuous driver action recognition in untrimmed videos that leverages large pretrained vision-language models. Firstly, the framework transforms every sampled frame into a natural-language caption, filters the captions with a text-only language model to extract distracted action evidence and core frames for video representation. Secondly, an open-vocabulary detector then spatially localizes the bounding boxes corresponding to the driver and several task-relevant objects on each core frame. Then, these bounding boxes together with the set of core frames are fed forward to a two-stage Chain-of-Thought (CoT) reasoning model to infer an action class for each core frame based on task-relevant visual evidences. Finally, a temporal post-processing is applied to refine and extend frame-level recognition to continuous recognition presented by a predicted action tube. Experiments on the UTCDriverAct dataset show that HybridLVLM attains a Video-mAP of 52.0% and a mean F1-score of 61.4% at an STT-IoU threshold of 0.3 without any training on driving data. These findings point to a promising research direction in video understanding for applications without real-time constraints. Additionally, an ablation study on the input sampling rate is also conducted to investigate the effect of the sampling rate on the overall recognition performance. The results indicate that the optimal sampling density depends on action duration, with higher rates for rapid actions and lower rates for gradual actions