This study aims to construct and validate a retrieval-augmented generation (RAG)-driven workflow for automatically generating SJT items and provides preliminary evidence for the feasibility of an automated development pathway for psychological assessment tools based on LLMs and RAG technology.
Abstract
In the field of civil aviation, the psychological competency characteristics of cabin crew members are directly related to service quality and flight safety. Although situational judgment tests (SJTs) have proven to be an effective assessment method, their development is costly and time-consuming. The breakthroughs in large language models (LLMs) offer new opportunities for the automated development of assessment tools. Using verbatim transcripts from critical incident interviews with frontline flight attendants as the primary data source, this study aims to construct and validate a retrieval-augmented generation (RAG)-driven workflow for automatically generating SJT items. An expert evaluation approach was employed to assess the quality of items generated by three large models (Model 1: qwen3-14b; Model 2: qwen3-32b; Model 3: deepseek-r1-32b). The results provide preliminary evidence for the feasibility of an automated development pathway for psychological assessment tools based on LLMs and RAG technology, which can significantly improve item development efficiency. However, this study represents an initial exploration, and further research as well as validation through large-scale empirical data are required to optimize and enhance model performance.
Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing assistants. General purpose benchmarks do not measure whether a model reasons safely and correctly about aviation specific operational knowledge, and the high stakes, regulated nature of the domain makes that gap consequential. We present Pre-Flight, an open source benchmark of 300 multiple choice questions drawn from international standards and airport ground operations material, covering international airport ground operations, ICAO and US FAA regulations, aviation general knowledge and complex operational scenarios. Questions were authored and reviewed by practitioners with experience in air traffic management, ground operations and commercial flying. We evaluate a range of contemporary commercial and open weight models using the Inspect evaluation framework, scoring by accuracy under a standard multiple choice protocol, and we maintain the leaderboard on a rolling basis as new models are released. Against an informal expert reference of around 95%, obtained from a low sample quiz of aviation professionals at a conference, even the strongest model evaluated (released in 2026) reaches 82.7%, having improved only gradually from roughly 75% in early 2025. A substantial and persistent gap below expert level reliability therefore remains. We release the dataset, the evaluation harness and the results, and the benchmark is available within the community evaluations package distributed with inspect_evals. We argue that domain specific evaluation of this kind is a necessary precondition for responsible deployment of generative AI in non safety critical aviation operations.
The correct regulatory interpretation in naval environments is challenging due to the complexity and urgency of decisions based on the International Regulations for Preventing Collisions at Sea (COLREGs). This article presents the development of an intelligent agent named Cognitive Agent for Analysis of Interrelated Problems in Maritime Navigation, hereafter referred to as CAPTAIMN. This agent integrates Large Language Models (LLM) and a Retrieval-Augmented Generation (RAG) architecture to support human decision-making and officer training in safety-critical naval environments. Thus, this work aims to propose a methodology for building an intelligent agent based on LLM and RAG, specifically focused on the assisted and contextualized interpretation of COLREGs. The proposed methodology was evaluated through a quantitative study with 15 maneuvering officers, who assessed 150 responses generated by a local language model using a Likert scale. The results from this phase showed significant approval, with 80% of the responses being rated as 'Agree' or 'Totally Agree' by the officers. These results suggest that the integration of LLM and RAG through CAPTAIMN can provide useful support for both decision-making and tactical training in naval operations.
Gabriel de Sapienza Luna, Arthur Pinheiro de Araújo Costa, Allyson A. da Silva et al.· International Conferences on...· 0 citations
Job Safety Analysis (JSA) and pre-task planning can benefit from prior incident records, yet historical accident data is often stored as unstructured narratives that are difficult to consult at the point of planning. A novel framework centered on large language models (LLMs) for highway construction safety reporting and planning is proposed as a foundation for future agentic applications, prioritizing deterministic, local inferencing. The first aim is to enable classification and quality scoring of incident narratives for existing and future reporting purposes. The second is to evaluate retrieval of relevant historical accidents, related imagery, and trusted industry documents for incorporation into daily safety plans. Neural probes were trained to classify incidents along four multiclass and two binary Occupational Injury and Illness Classification System (OIICS) fields and to derive an overall quality score, evaluated on a test set of over 15,000 narratives and a held-out set of 100 author-labeled records, benchmarked against a majority-vote LLM ensemble. The retrieval of historical accidents, reference imagery, and industry documents was benchmarked across embedding models using standard information retrieval metrics. OIICS classification reached 75% held-out accuracy, though the two binary flags were degenerate. The quality score, while meaningful on one database, was distorted on out-of-distribution fatalities in the held-out dataset. Accident retrieval recovered relevant incidents far above chance, performing best on lexically distinct construction activities. On document question answering, an open-weight decoder embedding model surpassed proprietary models. Overall, this work provides a new framework rooted in local inferencing and text embedding models for future agentic applications, with emphasis on bridging external data to JSA reports.
M. Smetana, Trevor Neece, Lev Khazanovich· 0 citations
Operational hazard analysis of aviation system operations must consider interactions among weather, ATC actions, airspace constraints, aircraft operations, and human factors - distinct from the functional hazard assessment applied at the aircraft-system level. We present an AI-assisted approach that generates candidate hazard scenarios from NASA's Aviation Safety Reporting System (ASRS). Given a target adverse outcome, it produces a structured hypothesis as categorical factors and a narrative scenario describing an operational event sequence consistent with the structure. Each scenario includes by a plausibility score from historical co-occurrence evidence and traceability to the most similar held-out ASRS reports. We then propose a hybrid variant, conditioning narrative generation on a structured hypothesis produced via evolutionary abduction, improving correctness and reducing variability. We evaluate multiple large language models, zero-shot versus few-shot prompting, and optional fine-tuning, measuring how prompting and model choice affect the validity and realism of the generated structures and narratives.
Cristian Mascia, R. Pietrantuono, Daniel Rodríguez et al.· 0 citations
Complex flight maneuver recognition (FMR) underpins intelligent flight training, including training assessment, pilot skill profiling, and flight safety monitoring. Existing FMR methods typically require large labeled datasets, generalize poorly across aircraft, and provide limited decision transparency. We propose TableManeuver, an explainable LLM-based FMR method that reformulates multivariate flight parameter time series as table-understanding inputs. The method updates no base LLM parameters and uses a small labeled training set only as a retrieval library; it is therefore not a zero-shot setting. TableManeuver first converts flight parameter sequences into tabular text that preserves temporal indices and channel semantics, reducing the mismatch between numerical time series and the textual semantic space of LLMs. It then combines domain knowledge, neighborhood sample references, and task decomposition prompts in a retrieval-augmented reasoning architecture that guides explicit step-by-step inference. We evaluate the method on a flight dataset collected from human pilots on a high-fidelity flight simulation platform. Without base LLM parameter updates, TableManeuver achieves 96.2% precision, 96.8% recall, and a 96.5% F1 score, exceeding the strongest supervised baseline by 3.5 percentage points in F1. In cross-aircraft evaluation, the F1 score decreases by only 1.4 percentage points, which is substantially smaller than the degradation observed for deep learning baselines. Retrieval-only baselines that transfer neighbor labels without LLM inference perform markedly worse, indicating that the performance gains are not explained by neighbor label transfer alone. TableManeuver combines recognition accuracy, cross-aircraft robustness, and readable step-by-step reasoning evidence, offering a practical route for applying LLMs to aviation time series analysis.
Liqiang Ren, Haipeng Wang, Xinlong Pan et al.· Entropy· 0 citations
Improving flight safety with flight data requires not only accurate detection of risk events, but more importantly, clear interpretation of their underlying causes at the level of pilot control behavior. Existing explainable AI techniques, such as feature importance maps, often require considerable domain knowledge to translate them into operationally meaningful explanations. Large Language Models (LLMs), which excel at language reasoning, bring a promising solution to this issue. However, applying LLMs in this domain presents key challenges such as modal inconsistency, limited classification ability, scarcity of task-specific data for fine-tuning, and lack of domain knowledge. To overcome these challenges, we propose FlightLLM, a prior-guided semantic LLM-based approach for interpretable flight safety analysis. Specifically, we first perform feature engineering to address modal inconsistency, combining statistical descriptors with physically meaningful flight indicators. This representation is further processed by a Semantic Discretization module, which converts abstract numerical patterns into qualitative descriptions that are more compatible with language reasoning. In addition, since LLMs are not inherently strong classifiers, CatBoost is incorporated as a statistical expert, and its prediction results are injected into the prompt as prior guidance. A contrastive few-shot learning strategy is further adopted to compensate for limited data. Finally, we design structured prompts to embed aviation-specific knowledge into the inference process. Using hard landing, a representative risk event with complex causal mechanisms, as an anchor point, we evaluate FlightLLM on a dataset of 704 real-world A320 flight samples. Experimental results show that the proposed approach achieves competitive classification performance while generating direct and reasonable explanations for event causes.