This work proposes a proof-of-concept pipeline that delivers on building an AI system that can ingest multi-modal data for railway crossings and provide safety assessment and scores that align with expert opinion and with safety scoring used by the Federal Railroad Administration.
Abstract
Given one or more images of a railway crossing, can we leverage visual cues that allow us to robustly estimate how safe it is? Can we improve our ability to do so by introducing structured data (such as official accident reports) about the accident history of that crossing into our models? In this work, we explore how to best answer those questions towards building an AI system that can ingest multi-modal data for railway crossings and provide safety assessment and scores that align with expert opinion and with safety scoring used by the Federal Railroad Administration (FRA). To that end, we propose a proof-of-concept pipeline that delivers on that goal, while at the same time exploring and tackling a number of critical research challenges that pertain to different parts of the pipeline, from data preparation to different learning paradigms that can allow us to realize such a system. Indicatively, our proposed system identifies HIGH-RISK and LOW-RISK crossings with a macro F1 score of 0.757 and estimates FRA-based safety scores with an RMSE of 0.078 and correlation of 0.492 using a routed fine-tuned compact VLM pipeline, while producing qualitative results that align with domain-expert assessment.
Job Safety Analysis (JSA) and pre-task planning can benefit from prior incident records, yet historical accident data is often stored as unstructured narratives that are difficult to consult at the point of planning. A novel framework centered on large language models (LLMs) for highway construction safety reporting and planning is proposed as a foundation for future agentic applications, prioritizing deterministic, local inferencing. The first aim is to enable classification and quality scoring of incident narratives for existing and future reporting purposes. The second is to evaluate retrieval of relevant historical accidents, related imagery, and trusted industry documents for incorporation into daily safety plans. Neural probes were trained to classify incidents along four multiclass and two binary Occupational Injury and Illness Classification System (OIICS) fields and to derive an overall quality score, evaluated on a test set of over 15,000 narratives and a held-out set of 100 author-labeled records, benchmarked against a majority-vote LLM ensemble. The retrieval of historical accidents, reference imagery, and industry documents was benchmarked across embedding models using standard information retrieval metrics. OIICS classification reached 75% held-out accuracy, though the two binary flags were degenerate. The quality score, while meaningful on one database, was distorted on out-of-distribution fatalities in the held-out dataset. Accident retrieval recovered relevant incidents far above chance, performing best on lexically distinct construction activities. On document question answering, an open-weight decoder embedding model surpassed proprietary models. Overall, this work provides a new framework rooted in local inferencing and text embedding models for future agentic applications, with emphasis on bridging external data to JSA reports.
M. Smetana, Trevor Neece, Lev Khazanovich· 0 citations
This paper proposes a novel level crossing (LC) control architecture designed for railway systems operating under ERTMS Levels 2 and 3. The approach aims to prevent identified hazardous scenarios and to facilitate the integration of LCs into the ERTMS/ETCS framework by moving from passive control schemes to a supervised, communication-based paradigm. The proposed strategy specifically addresses two critical issues: excessive level crossing closure time (ELCCT) and overly short opening durations (OSOD), both of which may lead to unsafe road user behavior. A formal behavioral model based on Time Petri Nets (TPN) is developed using a modular approach to represent the interactions between LC control and railway traffic. The model is enriched with observers to express safety requirements as temporal logic properties, which are verified using model-checking techniques with the TINA tool. The results show that the proposed architecture effectively prevents the considered risky scenarios.
Mohamed Ghazel· International Conference on...· 0 citations
Abstract. The safety, efficiency, and longevity of global railway networks are directly linked to the rigorous inspection and management of their vast inventory of physical assets. Over the past decade, the field has progressed from manual surveys to automated systems leveraging imagery from track-based or aerial platforms. These systems predominantly built on traditional Computer Vision (CV) models have proven effective at detecting a pre-defined set of common assets. However, this progress has exposed a fundamental architectural and operational ceiling: the closed-world assumption. Current models are constrained to a fixed catalogue of classes defined during their training. It makes the model incapable of identifying novel objects or adapting to environmental changes without costly and continuous cycles of data re-annotation, retraining, and redeployment. This review paper argues that Vision-Language Models (VLMs), a paradigm whose rapid maturation is evidenced by recent comprehensive surveys offer a transformative solution. We provide a focused overview of the limitations of current CV systems and map the mechanics of a VLM-powered approach specifically Open-Vocabulary Detection and Reasoning Segmentation directly to the outstanding challenges in rail asset management. Ultimately, the literature suggests that the adoption of VLMs could catalyze a fundamental shift in railway infrastructure management that serves as a key enabler for next-generation Predictive Maintenance and autonomous Digital Twins.
Ashley Varghese, Mohammadjavad Ghorbanalivaki, Gunho Sohn· The International Archives o...· 0 citations
Monitoring sea area use is essential for maritime safety, harbor management, and navigation planning. While the use of Automatic Identification System data has been studied extensively for trajectory-based tasks such as prediction, imputation, and collision prevention, such studies focus on individual vessel movements and do not provide area-level overviews of sea use—a key requirement for stakeholders such as harbor authorities and dredging agencies. We present the Seagull system for data-driven multi-level maritime traffic analysis. This system enables analyses of 16.4B AIS records from 101K vessels via a unified grid framework spanning from $40 \times 40 \text{km}$ cells, enabling regional traffic analyses, down to $2.1 \times 2.1 \mathrm{m}$ cells, enabling harbor-level analyses. By integrating with bathymetric depth models, the system supports safety-critical analyses involving under keel clearance, enabling identification of low-clearance zones and dredging needs. Employing a DuckDB star schema data warehouse for data storage, the system enables interactive analyses without pre-aggregation. Seagull is available online 11https://seagull.app.cs.aau.dk/.
Christian S. Jensen, Hengyu Liu, Kasper F. Pedersen et al.· International Conference on...· 0 citations
Transportation networks are critical for emergency response after earthquakes, but national-scale bridge and viaducts inventories often lack vulnerability-related attributes such as material and structural system. This paper presents an image-based approach, developed within the SAFENET project, to automatically classify bridges/viaducts according to a practical Material-Structure (MS) labeling scheme that reduces sparsity compared to finer taxonomies that also include construction period. Using a Portuguese bridge image dataset, we compare three visual model families for Material-Structure classification: a ResNet-50 convolutional baseline, a self-supervised vision transformer (DINOv2-Large), and a contrastive vision encoder (CLIP). Models are evaluated with a strict 5-by-5 Nested CrossValidation (NCV) protocol with bridge-level splits to prevent information leakage across train and test sets. Results show that DINOv2 achieves the best overall performance, with a mean accuracy of 0.903, a macro-F1 of 0.773, and a weighted-F1 of 0.897, outperforming ResNet-50 and CLIP especially on minority classes. These findings support the use of self-supervised vision transformers to enrich bridge inventories from imagery and to provide scalable inputs for regional seismic risk assessment.
Tomás Oliveira, Rui S. Moreira, Feliz Gouveia et al.· International Conference on...· 0 citations
Highway roads intersect with a railroad track, forming a multimodal transportation system. The intersection is called a highway railroad grade crossing (HRGC). Because of the multiple transportation operations, induced conflict at the grade crossing, crashes may happen between road users (e.g., automobiles, pedestrians, bicyclists, etc.) and trains in operations at highway railroad grade crossings. This is a global challenge for many countries around the world. According to the 2022–2026 Indiana Strategic Highway Safety Plan and 2022 Indiana Highway-Rail Grade Crossing Safety Action Plan, one of the biggest challenges facing Indiana is the higher numbers of pedestrians in and around crossing locations and the possible eligibility of Federal funds to address nonmotorized safety concerns. This study systematically examined the grade crossing safety impacts among human factors, environmental factors, warning device conditions, and other factors through extensive site visits and data analysis. Key findings indicate that grade crossing safety is a function of multiple parameters. Different factors have various levels of contribution to the potential crashes. Crash report analysis and site visit outcomes consistently highlighted some factors’ impacts on the grade crossing safety for nonmotorized users in Indiana. Implementing the insights from this study is expected to improve grade crossing safety, mitigate potential grade crossing crashes, and optimize the budget for grade crossing upgrades and improvements.
Shanyue Guan, Siqi Chen, Shufan Liu et al.· 0 citations