The proposed Multimodal Safety Framework aimed at analyzing images of work environments at heights to identify potential risks and recommend preventive measures provides a viable and scalable solution for significantly improving OHS compliance and mitigating risks in complex working environments.
Abstract
In this work, we address the critical challenge of Occupational Health and Safety (OHS) inspection for work at heights, a task that requires simultaneous visual analysis and extensive regulatory knowledge. We propose and implement a Multimodal Safety Framework aimed at analyzing images of work environments at heights to identify potential risks and recommend preventive measures. The primary novelty of this work lies in the system integration of these tasks into a unified pipeline, which connects advanced object detection techniques, with Multimodal Large Language Models (MLLMs). The object detection component is responsible for localizing and classifying safety-critical entities, such as personnel, Personal Protective Equipment (PPE), and hazards (e.g., unstable ladders or scaffolding), within the visual scene. The MLLM then processes these visual cues alongside implicit regulatory knowledge to generate a high-level risk assessment and suggest prescriptive preventive actions. The proposed framework was evaluated on a custom dataset of images depicting various high-risk scenarios in construction. Results demonstrate that the system achieves high accuracy and formulation of relevant, context-aware safety recommendations. This integration indicates that the proposed combination of computer vision and multimodal architectures provides a viable and scalable solution for significantly improving OHS compliance and mitigating risks in complex working environments.
This study formally evaluates the effectiveness of AI models over multiple iterations of the models’ architecture for the domain-specific application of automated construction hazard assessment from multimodal inputs and introduces and validates high-fidelity, game engine-based synthetic images as a solution.
Trevor Neece, A. Fascetti· Journal of computing in civi...· 1 citation· ⚡1
The primary objective of this study is to assess the effect of systematic dataset augmentation on the accuracy of real-time, vision-based Personal Protective Equipment (PPE) detection systems in occupational environments. The PPEDS-1000 dataset was employed, comprising 1,000 expertly annotated images across four PPE usage categories: worker (W), worker with helmet (WH), worker with vest (WV), and worker with both helmet and vest (WHV). An augmented dataset (PPEDS-2600) was derived via controlled geometric transformations (horizontal and vertical flips), additive Gaussian noise, and Gaussian blur. Each dataset is partitioned using an 80/10/10 train–validation–test split and utilized to train five YOLOv8 model variants (nano through extra-large). The evaluation metrics include precision, recall, F1-score, mean average precision at an IoU threshold of 0.5 (mAP50), and mean average precision averaged over IoU thresholds from 0.5 to 0.95 (mAP50-95). The experimental results demonstrate that augmentation elevates mAP50 from 77.7% on PPEDS-1000 to 94.8% on PPEDS-2600, thereby substantiating the hypothesis that targeted augmentation markedly enhances detection performance. The findings indicate that the present work establishes a rigorous benchmark for real-time PPE compliance monitoring.
Öyküm Akar, Hasan Selim, Orhan Er et al.· Intelligenza Artificiale· 0 citations
Human Activity Recognition (HAR) is the automated process of identifying human actions using sensor data or video, which is widely used in healthcare, smart environments, and surveillance. Although HAR based on computer vision has advanced rapidly, existing reviews do not adequately address the recent shift toward hybrid deep-learning architectures or provide a structured comparison of the trade-offs relevant to real-world deployment. This literature review addresses that gap through a PRISMA-guided analysis of articles published between 2021 and 2025 and retrieved from four major databases. The review develops a reproducible taxonomy of nine architectural families and applies a multidimensional evaluation framework covering classification accuracy, computational efficiency for edge deployment, environmental generalization, and fine-grained activity recognition. The findings show that hybrid architectures are the dominant design strategy, while attention-based and graph-based models play important specialized roles depending on temporal complexity, privacy requirements, and deployment constraints, with the literature concentrated mainly in healthcare and security applications.
Timely and accurate recognition of workplace hazards is critical for ensuring safety in dynamic and high-risk environments such as construction sites. However, existing video-based approaches often rely on extensive annotations and fully supervised training, and many are tailored to specific hazard types, which limits scalability to rare or diverse scenarios. This study investigates whether a data-efficient vision–language framework can support early-stage hazard recognition from short preincident video segments under limited supervision. The proposed method builds on a video-adapted contrastive language-image pre-training (CLIP)-based vision–language backbone and introduces a parameter-efficient adaptation strategy that combines learnable visual prompting with low-rank adaptation (LoRA) applied to the text encoder. Learnable visual prompts capture global, summary, and local spatiotemporal cues from short video clips, and LoRA enables lightweight semantic adaptation without fine-tuning the full backbone. This design preserves most pretrained parameters and introduces only a small number of additional trainable parameters, making it well suited to few-shot learning in data-scarce safety scenarios. To evaluate early recognition capability, this paper further examines performance under shorter temporal observation windows by reducing the amount of video available for prediction. Experiments on a curated real-world hazard video data set show that the proposed method substantially improved few-shot recognition compared with a prompt-free baseline across five hazard categories and 5-, 10-, and 15-shot supervision. Overall, the findings suggest that combining temporal visual prompting with parameter-efficient text adaptation offers a scalable engineering pathway toward assistive hazard recognition in data-scarce environments, with promising but preliminary performance.
Masoud Ayoubi, M. Arashpour· Journal of construction engi...· 0 citations
Workplaces in construction and industry suffer from a significant number of workplace accidents because of a lack of safety mechanisms like helmets and high-visibility vests. In order to solve this issue, an automatic detection system for detecting the presence of the worker's helmet & safety vest using a deep learning model is developed. You Only Look Once – Neural Architecture Search (YOLO-NAS) algorithm was chosen to be used in the model because of its fast and high-quality detection process. A dataset with images containing healthy workers wearing helmets & safety vests (with some workers appearing in different safety vests) is collected and used as the training set. The resulting model is then saved for future usage in making predictions with new images. Then, this model is connected to Streamlit, which provides a convenient Web-based user interface through which a user can provide an image as an input to the model. The trained neural network is applied to the input image, and all detected objects that correspond to helmet & safety vest are put in bounding boxes on the input image. Moreover, each of the bounding boxes is labeled with the name of the object and confidence score of the detection. Thus, a clear visualization of the safety equipment worn by the worker is provided. Overall, the developed detection system significantly decreases the human intervention needed for visual verification of the safety helmet or vest. The described system is a user-friendly and cost-effective way of evaluating workplace safety compliance.
S. Vijayakumar, Loganathan Nachimuthu, Balasubramaniam C et al.· 2026 International Conferenc...· 0 citations
A visual knowledge enhancement framework for construction OHS visual question answering (VQA) based on multimodal large language models (MLLMs) and retrieval-augmented generation (RAG) is proposed, augmenting managerial capacity for reliable and objective OHS hazard prevention.
Yang Liu, Luping Li, Xing Su et al.· Journal of Management in Eng...· 0 citations