Automatic traffic surveillance system leveraging generative AI large language models
TL;DR
A generative AI-based traffic surveillance system leveraging large language models (LLMs) to enable timely and context-rich interpretation of traffic events, demonstrating the system's effectiveness in producing context-aware traffic scene descriptions, improving operational decision-making, and enhancing roadway safety.
Abstract
The application of automated traffic surveillance systems has become increasingly critical for improving traffic management and reducing reliance on manual monitoring. Traditional video surveillance methods are time-consuming, resource-intensive, and prone to human error, with operators frequently missing incidents due to fatigue and environmental factors. This study introduces a generative AI-based traffic surveillance system leveraging large language models (LLMs) to enable timely and context-rich interpretation of traffic events. We developed a custom-annotated dataset of 604 videos, capturing crashes, congestion, lane closures, and diverse weather and lighting conditions from Missouri Department of Transportation cameras and online sources. This approach fine-tunes the Qwen2.5VL-Instruct model using Low-Rank Adaptation (LoRA), temporal context enhancement, and multidimensional Rotary Position Embedding (mRoPE) for improved cross-modal fusion. Compared with the LLaVA-NeXT-Video baseline, the fine-tuned model achieves substantial gains in captioning performance (BLEU-4 = 0.4569, METEOR = 0.6187, CIDEr = 5.4612) and reduces average video description time from 35 seconds (manual) to 20 seconds (automated). An interactive web interface integrates real-time traffic monitoring with automatic incident detection and visualization, supporting faster emergency response and scalable deployment across traffic management centers. These results demonstrate the system's effectiveness in producing context-aware traffic scene descriptions, improving operational decision-making, and enhancing roadway safety.