Can an LLM Agree with Itself? Repeatability and Reproducibility in Grant Screening

Abstract

Large language models are increasingly being considered for grant proposal screening, yet their reliability has received less attention than their agreement with human reviewers. This case study evaluates the repeatability and reproducibility of direct textual interpretation by DeepSeek-V4-Flash-0731, run in reasoning mode, on 1,026 applications to a 2026 science and technology project program. Seven categorical questions were evaluated under three conditions corresponding to three prompt versions, with ten runs per application and 30,780 model calls in total. Reliability was measured as pairwise response disagreement at the application level. Cross-condition drift was adjusted for each condition’s intrinsic stochastic drift. Under identical settings, mean intrinsic drift was 8.87% across questions, and only 15.8%of applications were fully stable on all seven questions. Semantic prompt reformulation produced effects more than an order of magnitude larger than spelling and punctuation correction and shifted response probabilities by over 50 percentage points for some questions. In one limiting case, adding a single comma changed the share of affirmative responses from 31.8% to 48.8% without changing any words. These findings suggest that even direct textual interpretation should be delegated to an LLM only after questions have been decomposed, carefully worded, and empirically tested for reproducibility.