Back to feed
Conference

SafetyBuddy: A Multimodal LLM-Based Safety Intelligence Platform for Regulatory Compliance in Process Industries

Aug 2026 · SPE Nigeria Annual International Conference and Exhibition · 0 citations · 18 references

TL;DR

The problems addressed in this paper are platform architecture, multimodal data fusion and LLM grounding mechanism in the Nigerian industrial system in addition to evaluation metrics, security control, human-in-the-loop validation of safety alerts and the limitations to real-world deployment.

Abstract

Process industries are a major source of occupational injuries and fatalities worldwide due to safety violations. Nigeria alone has recorded more than 412 fatalities in the oil and gas industry and the construction industry has a documented history of accidents associated with personal protective equipment (PPE) non-compliance. According to the World Risk Poll (2024), Sub-Saharan Africa is the highest-ranked region in terms of workplace injuries, where 21% of the existing workforce experiences serious harm. The currently used compliance monitoring methods are largely manual, ad hoc, and not linked with regulatory frameworks. This paper describes SafetyBuddy, a multimodal safety intelligence application designed to automate compliance monitoring. The system incorporates a YOLO26-nano real-time object detector, retrieval augmented generation (RAG) over OSHA regulatory documents, and reasoning on a continuous PPE compliance system powered by the open-weight Gemma 4 multimodal large language model (LLM), self-hosted on serverless GPU infrastructure. Its functionality spans four states of interaction, including safety advisory, incident root cause analysis, compliance auditing, and real-time video violation alerting. After 50 training epochs, the YOLO26 model obtained mean average precision (mAP@50) of 0.691 and precision of 0.775 on a 10-class dataset, with NMS-free inference at 25-40 frames per second on CPU hardware. All recommendations are enhanced with OSHA regulations traceability by an independent compliance mapping module. Running on scale-to-zero serverless infrastructure and costing around USD 10-15 each month, the platform proves that a resource constrained industrial setting can have perception, regulatory knowledge and reasoning that are united to enforce safety compliance. The problems addressed in this paper are platform architecture, multimodal data fusion and LLM grounding mechanism in the Nigerian industrial system in addition to evaluation metrics, security control, human-in-the-loop validation of safety alerts and the limitations to real-world deployment.

View source

Similar papers

Dependable Person Detection using AI in Industrial Environments

Suggestions for ensuring safe person detection using AI in industrial environments are offered, including suggestions for ensuring safe person detection using AI in industrial environments.

Iwo Kurzidem, Andrea Matic-Flierl, Poulami Sinhamahapatra et al. · 0 citations
Open access 2026

Implementing Artificial Intelligence to Reduce Risks in Auto Repair Services

Auto repair workshops are high-risk environments where workers face frequent accidents caused by slips, trips, heavy lifting, exposure to hazardous substances, and malfunctioning equipment. Despite advances in occupational health and safety (OHS) regulations, accident rates in Romania and across the European Union remain above average, highlighting the need for new preventive strategies. This paper explores how artificial intelligence (AI) can be systematically applied to reduce operational and safety risks in auto repair shops. Accident statistics and a detailed risk assessment of slips, trips, and falls provide the foundation for analyzing AI-based solutions, including computer vision, augmented reality, smart floors, voice assistance, and predictive analytics. A six-month pilot project conducted in a Romanian auto repair shop demonstrated a 28% reduction in minor accidents, faster hazard detection, and measurable economic benefits, while also revealing challenges such as false positives, resistance to change, and maintenance requirements. The findings confirm that AI can move workplace safety from reactive to preventive management by enabling real-time monitoring, proactive alerts, and predictive forecasting. The study concludes that AI, when combined with training and organizational adaptation, can significantly enhance worker protection, operational efficiency, and sustainability in the automotive repair sector.

A. Cană, Claudia Borda, Adrian Moise et al. · 0 citations

Comparative Analysis of Multimodal AI Models for Automated Construction Safety Monitoring and Reporting

This study formally evaluates the effectiveness of AI models over multiple iterations of the models’ architecture for the domain-specific application of automated construction hazard assessment from multimodal inputs and introduces and validates high-fidelity, game engine-based synthetic images as a solution.

Trevor Neece, A. Fascetti · 1 citation · ⚡1
Aug 2026

SafeRAG: A Large-Language-Model-Based Multistage Retrieval-Augmented Framework for Oil and Gas Safety Report Generation

In high-risk industrial settings, leveraging large language models (LLMs) for automated accident analysis and generating safety reports has emerged as an efficient workflow. However, this approach is fundamentally constrained by the models’ inherent knowledge limitations, frequently resulting in analyses that lack domain-specific understanding and regulatory alignment. To tackle this issue, we introduce SafeRAG, a multistage retrieval-augmented framework for safety report generation. Specifically, the framework uses an entity-centric approach that prompts the LLMs to internally generate domain-specific knowledge. Concurrently, it performs a hierarchical retrieval of external regulations relevant to the accident at topic, concept, and context levels. To obtain well-structured reports, we leverage prompt engineering, integrating internal and external knowledge. Furthermore, a domain-expert persona is also assigned to help LLMs analyze accidents from a specific perspective. To evaluate our approach, we construct a data set from 10,818 accident-description/report pairs collected from real-world industry reports. Experiments show that SafeRAG substantially outperforms baseline LLMs on metrics that include bidirectional encoder representations from transformers (BERTScore) and bidirectional auto-regressive transformers (BARTScore), demonstrating the effectiveness of our approach.

Qi Cao, Chiyu Zhang, Junxiong Ning et al. · 0 citations
Open access Jul 2026

Key Technologies and Applications of Multimodal Cognitive Agent Construction in the Vertical Domain of Inspection and Testing

 Inspection and testing underpin product safety and regulatory compliance across industries such as manufacturing, healthcare and food and beverage. However, conventional engineering test processes that mainly depend on manual, offline actions, failed to adapt to next generation, automated, digital and environmental-sensitive testing, resulting in very low productivity and high cost for most firms. The automation and digitalization of inspection and testing processes have become a research hotspot in both academia and industry. The natural language processing and computer interpretation of test results have been a focus of AI research as well. Nonetheless, due to the lack of real data, the verification and simulation of real inspection and testing environments are still difficult for researchers. As a solution, we manage to develop a multimodal cognitive test system that fuses the textual regulatory documents and instrument data, through a series of modules and processing flows. Our test system is based on a so-called multimodal cognitive agent, which includes large language model, vision module, knowledge graph and retrieval-augmented generation. We introduce the design, development and application of our test system, which used for rubber heater in a glass factory, and layout the future challenges for the exploration of multimodal agent technology in the test engineering. The project is supported by the Guangxi Key Research and Development Program. This paper was partially presented at the IEEE 2023 International Conference on Intelligent Commerce (ICIC).

Zhiruo Li, Wenlin You, Xinrong Zhang et al. · 0 citations
Preprint Aug 2026

SafeSceneReason: A Multimodal Reasoning Benchmark Connecting Industrial Hazards with Accident Knowledge

Industrial-safety understanding requires more than detecting workers, equipment, and personal protective equipment. Models must also assess compliance, identify hazardous interactions, explain potential accident mechanisms, and recommend preventive actions. Existing safety datasets primarily focus on visual perception or isolated violation recognition and provide limited supervision for evidence-grounded reasoning. We introduce SafeSceneReason, a multimodal industrial-safety reasoning benchmark and companion training corpus that connects workplace scenes with knowledge from occupational accident investigations. SafeSceneReason combines two complementary data-construction pipelines. The scene-centric pipeline converts annotated workplace images into executable safety scene graphs and generates deterministic answers through program execution over objects, relations, and safety rules. The report-centric pipeline extracts figures and contextual evidence from accident reports and constructs multimodal questions using evidence graphs, explicit information boundaries, multi-step reasoning paths, and iterative verification. The resulting resource contains 110,581 verified scene-centric question--answer pairs and 13,114 refined report-centric question--answer pairs, covering perception, spatial and quantitative reasoning, compliance assessment, evidence synthesis, causal analysis, and mitigation-oriented decision making. Evaluation of representative proprietary and open-source vision--language models reveals substantial performance differences and persistent weaknesses in comparative, technical, and multi-evidence reasoning, demonstrating that strong general visual understanding does not yet guarantee reliable industrial-safety reasoning.

Yuanchi Zhu, Kang An, Tengyue Wang et al. · 0 citations