Attend to Fragments: How Key Information Affects Large Language Models for Factual Inconsistency Detection

Abstract

As large language models (LLMs) continue to advance, a key challenge remains their tendency to hallucinate, generating fluent yet inconsistent content that lacks factual grounding. Natural language inference (NLI)-based methods, which determine whether one statement can be logically inferred from another, are widely considered the most effective for detecting input-output inconsistencies in LLMs. However, several fundamental questions, such as whether LLMs can identify relevant information to make correct factual inconsistency detections and how different arrangements of the source document affect reasoning, are not discussed in prior studies. To bridge this research gap, we design a new benchmark, KIFI, which comprises 1032 carefully selected instances from the TRUE and ScreenEval datasets, with key information annotated. Using KIFI, we show that LLMs frequently fail to use the appropriate information to make correct decisions. In addition, we find that LLMs tend to make predictions by overemphasizing certain keywords or fragments, a new phenomenon we term "Attend to Fragments". We further introduce a novel token-based permutation method to identify untrustworthy inconsistencies. Experiments show that filtering out these instances improves the overall correlation by 1.3% on the standard TRUE benchmark. The project is available at https://github.com/VibeHPC/attend-to-fragments