Survey-country metadata can improve an LLM's forecast of an individual response when informative, yet the same cue may redirect the forecast when assigned at random. A within-record audit tests whether disclosing a random label's uniform, record-independent origin reduces its country-directed uptake, and whether verified survey country lowers held-out Brier loss. Independent population anchors and recorded human answers measure direction and consequence across five fixed API models, six countries, and seven development-selected targets. In the primary post-review 72-record panel, opaque and disclosed-random labels each produced country-direction shifts of 0.214. Paired attenuation was 0.0003 (95% CI [-0.0157, 0.0166]). Verified country reduced Brier loss by 0.040 (95% CI [0.024, 0.056]), while random-label regret included zero. A non-overlapping mixed-coverage consistency panel retained positive disclosed-random movement and verified utility, while attenuation remained uncertain. On the selected targets, verified metadata was useful in both panels, but disclosure did not reliably attenuate random-label uptake. PROV-FORECAST contains 14,400 paired item-level probability distributions from the corrected panel.
Silicon sampling uses large language models (LLMs) to simulate survey respondents. Whether it recovers cross-national variation, and why, remains unresolved. This study evaluates it against European Social Survey Round 11 (30 countries, 42 items) with two open-weight LLMs under first- and third-person prompts, plus bac...
Generative search systems rank products and services for consequential decisions, and publishers can cheaply make candidate text look relevant. Yet evidence status is not a text property but a claim-evidence relation: text-only rankers and defenses cannot separate honest detailed content from fabricated detail, creatin...
Yi-Han Xia, Hui-Ling Fan, Kang-Rong Zhong et al.· 0 citations
These findings support calibrated LLM trust vectors as a potentially useful archival proxy while highlighting benchmark dependence, sampling constraints, model sensitivity and the importance of reproducible data provenance.
Pieter Claassen, G. V. Van Vuuren, Tanja Verster· Information· 0 citations
Whether a language model looks demographically biased can depend on how the audit asks its question. A charitable-aid benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side. We test whether that reversal generalizes to hiring, lend...
Retrieval-augmented fact-checkers often receive a reliability label, such as HIGH or LOW trust, for each evidence source. These labels should adjust the model's confidence and its decision to search for more evidence, while the verdict should follow the evidence content. We introduce TrustSwap, a counterfactual test th...
Jian-Chang Su, Yi-Wei Yang, Wei Zhang· 0 citations
Self-report financial surveys require auditable scoring as well as evidence that their items measure the intended constructs. This study combines a Peruvian workplace archive of 50 records with an independently reconstructed Monte Carlo experiment. The audit distinguishes archived categories, direct sums, a historical...
Fabricio Miguel Moreno-Menéndez, José Francisco Vía Y. Rada-Vittes, Nataly Gabriela Solis-Tapia et al.· Journal of Risk and Financia...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
