Leveraging Multimodal Artificial Intelligence for Occupational Health and Safety: A Deep Learning Approach for Work at Heights

Abstract

In this work, we address the critical challenge of Occupational Health and Safety (OHS) inspection for work at heights, a task that requires simultaneous visual analysis and extensive regulatory knowledge. We propose and implement a Multimodal Safety Framework aimed at analyzing images of work environments at heights to identify potential risks and recommend preventive measures. The primary novelty of this work lies in the system integration of these tasks into a unified pipeline, which connects advanced object detection techniques, with Multimodal Large Language Models (MLLMs). The object detection component is responsible for localizing and classifying safety-critical entities, such as personnel, Personal Protective Equipment (PPE), and hazards (e.g., unstable ladders or scaffolding), within the visual scene. The MLLM then processes these visual cues alongside implicit regulatory knowledge to generate a high-level risk assessment and suggest prescriptive preventive actions. The proposed framework was evaluated on a custom dataset of images depicting various high-risk scenarios in construction. Results demonstrate that the system achieves high accuracy and formulation of relevant, context-aware safety recommendations. This integration indicates that the proposed combination of computer vision and multimodal architectures provides a viable and scalable solution for significantly improving OHS compliance and mitigating risks in complex working environments.