A statistical framework for reliability evaluation of large language models

Abstract

The growing use of Large Language Models (LLMs) in real-world applications has increased the need for reliable evaluation methods. This study proposes a statistical framework for evaluating LLM reliability by jointly considering model accuracy, hallucination rate, and cross-domain performance stability. Based on the TruthfulQA dataset, 300 test questions were selected using a stratified sampling strategy, and responses were generated by DeepSeek-V4-Flash and GPT-4o-mini. GPT-4o was then employed as an automated evaluator to assess the 600 generated responses. Model reliability was evaluated using three dimensions: Accuracy, Hallucination Rate, and Cross-Domain Stability. In particular, the proposed CDS metric quantifies the consistency of model performance across different knowledge categories. A Weighted Geometric Mean was further employed to construct an overall reliability score. The experimental results show that the proposed framework can identify reliability differences between different LLMs. DeepSeek-V4-Flash achieved an overall reliability score of 0.770, compared with 0.678 for GPT-4o-mini. Sensitivity analysis further shows that the model ranking remains unchanged under different weight configurations, indicating that the proposed framework is reasonably robust to variations in indicator weights. This study provides a statistical framework for quantitatively assessing LLM reliability under multiple evaluation criteria.