Toward Sustainable Urban Mobility: A Multimodal Large Language Model (MLLM) Framework for Automated Driver Performance Assessment with YOLOv8-Based Scene Detection

Abstract

Accurate and scalable driver performance assessment is critical for improving road safety and reducing traffic-related injuries and fatalities, particularly in low- and middle-income countries where the majority of global road deaths occur. This paper presents an exploratory proof-of-concept framework for automated driver evaluation that combines real-world dashcam footage, YOLOv8-based object detection, and multimodal large language models (MLLMs), specifically Gemini 1.5 Flash. Two prompting strategies, narrative and rule-based, were designed to assess driver behavior against standardized licensing criteria derived from the California Department of Motor Vehicles (DMV) driving performance evaluation score sheet. The framework was evaluated across 11 manually curated driving scenarios covering intersections, pedestrian crossings, stop signs, cyclists, and emergency vehicles. Ground-truth labels were established through consensus between two traffic engineering experts cross-referencing official California DMV evaluation criteria. In this preliminary evaluation, the rule-based prompt achieved higher agreement with ground-truth assessments (10/11 scenarios, 90.9%) compared to the narrative prompt (7/11 scenarios, 63.6%), particularly in detecting clear rule violations. The narrative approach demonstrated greater contextual flexibility in ambiguous situations. These results should be interpreted as preliminary, given the small sample size, manually curated dataset, and absence of large-scale statistical validation. Nonetheless, the findings illustrate how combining visual detection with structured language-model prompting may support interpretable, policy-aligned driver evaluation. Key limitations include dependence on video quality, limited scenario diversity, absence of temporal behavioral modeling, and reproducibility constraints tied to proprietary API behavior. Future work should expand validation to larger annotated datasets, incorporate temporal sequence modeling, and explore region-specific regulatory adaptation.