These findings endorse the viability of LLMs as a feasible alternative for data error detection; nonetheless, challenges such as hallucination and cost aspects remain, especially regarding the resources necessary for the effective implementation of these models in practical applications.
Modern Software-as-a-Service (SaaS) environments generate voluminous, heterogeneous logs that exhibit rapid schema drift and semantic sparsity, rendering traditional template-based detection methods ineffective. This paper presents LLM-EP, a unified framework that utilizes Large Language Models (LLMs) for proactive error prediction via semantic log analysis. Unlike existing heuristics, LLM-EP employs a drift resilient semantic parser and an embedding driven normalization engine to maintain accuracy amidst evolving log formats. We propose a hybrid offline-online architecture that balances deep semantic understanding with real-time inference requirements, incorporating a robust outof-vocabulary (OOV) mechanism for zero-shot generalization to unseen log patterns. Evaluation on production SaaS datasets indicates that LLM-EP achieves a 25% gain in parsing accuracy and a 12% improvement in F1-score over state-of-the-art baselines. Most notably, LLM-EP reduces Mean Time to Detect (MTTD) by up to 60%, demonstrating its efficacy as a scalable foundation for automated site reliability engineering.
In industrial processes, the infrequent and unpredictable faults often lead to small sample sizes and a limited amount of labeled fault data. Traditional data‐driven methods struggle under these conditions, as they typically rely on large volumes of labeled fault data for effective training. To overcome these limitations, an exploratory fault detection method based on the pre‐trained large language model (LLM) is proposed. The pre‐trained LLM is utilized to extract fault features from small, imbalanced datasets without the explicit labels. Additionally, a stepwise tuple‐based validation process is introduced to ensure logical consistency in fault analysis and to mitigate the effects of potential LLM hallucinations. The proposed method is validated using a case study involving a sudden feedstock (coal) shortage fault in a circulating fluidized bed boiler. The results demonstrate an improvement in fault detection accuracy over conventional methods in this case study (achieving an accuracy of 0.948 and an F1‐score of 0.926), confirming the feasibility of the proposed approach. The viability of integrating LLM‐assisted fault detection into specific industrial scenarios is investigated in this work.
Yihua Ye, Yin Zhu, Liming Che et al.· Safety Science and Technolog...· 0 citations
Large Language Models are reshaping how software is developed and maintained. They are typically deployed in production using inference engines such as vLLM, which can efficiently serve pre-trained, highly configurable models. While prior work has focused on model architectures and hardware acceleration, the impact of inference engine configuration on energy consumption, performance, and output quality remains poorly understood. In this paper, we present a large-scale controlled study of three selected vLLM configuration options: attention kernel type, prefix caching, and chunked prefill. We evaluate all combinations of these configurations across 5 open-weight LLMs and 5 diverse inference tasks, totaling $9,000$ runs and $93,600$ measures. We analyze energy consumption, latency, and accuracy, and examine both main effects and interaction effects between configuration options and tasks. Our results show that the studied configuration options significantly impact energy and performance, mainly driven by attention type and prefix caching, while chunked prefill has a limited effect under the default vLLM serving configuration and evaluated workloads. These effects are highly model- and workload-dependent, and no configuration is universally optimal. We further show that model choice dominates global trade-offs, while configuration tuning provides local improvements along the Pareto frontier. Unexpectedly, inference options can also affect model accuracy.
Nada Zine, Tristan Coignion, V. Stoico et al.· 1 citation
The rapidly growing repository of publicly available large language models (LLMs) presents significant challenges for systematic management and quantification at scale, such as model lineage tracing, licensing, and evaluation. However, task-specific benchmarks are insufficient for this setting, as LLMs differ widely in architectures, scales, and training procedures. To address this challenge, we adopt spectral shape -based metrics for managing and quantifying LLMs based on Heavy-Tailed Self-Regularization theory. Our approach uses the shape information of the weight empirical spectral density as a compact spectral signature of each model. This signature captures intrinsic properties of pretrained models and remains robust during post-training, making it suitable for model-level analysis. In addition, this metric is data-free, computationally-efficient, and scale-invariant, enabling large-scale analysis in practice. Moreover, we curate a large and diverse model corpus consisting of major open-source LLM families, and use it to systematically benchmark spectral and non-spectral metrics across models and downstream tasks. We show that our spectral signature supports the tracking of the model lineage, the unsupervised clustering of similar models, and the quantification of the model performance. Overall, the proposed spectral signature provides a meaningful proxy for broad performance trends across LLMs, enabling efficient organization, comparison, and analysis of large model collections. The code is available at https://github.com/Ingrid-505/Spectral_Signature.
Zhuoying Zhang, Ishanvi Prasad, Yuanzhe Hu et al.· Proceedings of the 32nd ACM...· 0 citations
Anomaly detection in tabular data is challenging because abnormal samples often arise as violations of cross-feature dependencies rather than simple marginal deviations. Existing detectors rely on geometric or reconstruction signals, while prior LLM-based approaches mainly fine-tune LLMs with normal samples or generate synthetic anomalies. We propose LLM-Detector, a framework that utilizes the in-context learning capacity of LLMs for structured, prompt-conditioned scoring synthesis, enabling LLMs to derive anomaly detection logic from structured normal-state knowledge. Specifically, normal training data are converted into statistical summaries, causal dependencies, and distilled prototypes that are organized into a prompt for code generation. The resulting scoring engine evaluates statistical deviation, structural inconsistency, and density-based abnormality then computes an anomaly score for each test sample. We evaluate LLM-Detector on 24 tabular datasets, comparing against 15 SOTA baselines. Results show consistent improvements across both mixed-type and continuous-only settings. Moreover, this design eliminates the need for LLM fine-tuning or neural network training, reducing computational cost and enabling practical anomaly detection in real-world tabular systems.
Large language models are increasingly used to evaluate and support software engineering tasks, yet the validity of these evaluations is often undermined by uncertainty about whether benchmark instances were seen during pretraining. This can lead to data contamination, which may inflate performance and result in misleading conclusions about model capability. Despite this, the training corpora of many modern models are only partially disclosed, making direct decontamination infeasible. This creates a need for practical methods that can detect a large language models' prior exposure to training data without access to the full training corpus. To address this challenge, we organize the first Poisoned Chalice of LLM Evaluation Competition, co-located with the FSE-AIWare 2026 Competition Track. The competition frames contamination detection as a white-box membership inference task on source code and provides participants with curated datasets, target models, baseline attacks, and a final evaluation on a held-out model and dataset. This design encourages methods that generalize beyond superficial dataset artifacts and beyond a single training setting. This paper reports the setup and results of the competition. More broadly, the competition aims to catalyze the community around trustworthy LLM evaluation for software engineering.
J. Katzy, Ali Al-Kaswan, R. Popescu et al.· SIGSOFT FSE Companion· 1 citation· ⚡1