Skip to content

Count Evidence, Not Sentences: Tempered Evidence Fusion of LLM Judgments for Long-Text Value Measurement

Sep 2026 · 0 citations · 31 references
Computer Science

TL;DR

Tempered Evidence Fusion (TEF) is proposed, a training-free rule that weights each sentence's log-odds by its normalized information gain, as derived from a generalized Bayesian posterior, making the fused score nearly vanish for uncertain sentences while preserving the Bayes-optimal weight of decisive evidence.

Abstract

Large language models (LLMs) are increasingly used to measure public value orientations from long social media posts, yet such posts often mix background, quotations, concessions, and only a few stance-bearing sentences. Existing approaches either ask the model to predict a document-level label directly, which can be overconfident, or aggregate sentence-level predictions by majority or soft voting, which treat uncertain and decisive sentences as equally informative. We formulate long-text value measurement as a decision-fusion problem and propose Tempered Evidence Fusion (TEF), a training-free rule that weights each sentence's log-odds by its normalized information gain, as derived from a generalized Bayesian posterior. This makes the fused score nearly vanish for uncertain sentences while preserving the Bayes-optimal weight of decisive evidence. We further introduce Multi-event Insight Network Dimensions (MIND), a benchmark of 8,358 Chinese and English posts spanning five years of public events and six value dimensions. On MIND, TEF outperforms the strongest baseline among Direct, Majority Vote, and Soft Vote by an average of 4.5 accuracy points and 4.6 macro-F1 points across five LLMs and two languages. MIND dataset and code are available at https://github.com/Kzczc/ICASSP2027-TEF.

View source

Similar papers

#natural language process... Preprint Oct 2026

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different...

Laurène Vaugrante, Thilo Hagendorff · 1 citation
Preprint Nov 2026

Finding Errors Is Not Enough: Selective Model Escalation for Entity-Targeted Sentiment Analysis

Improving a sentiment classifier with limited resources involves more than simply deciding how much data to collect or which model to train. A labelled collection may lack examples of an important class. Once trained, a classifier may benefit from another model even when that model is weaker on its own. We study these...

Nishan Chatterjee, Matthew Purver, Antoine Doucet et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models

Multilingual evaluations often score a model's opening yes or no, a token that also carries model and language answering conventions. We ask when different ways of eliciting and reading an answer recover the same judgment, and when they measure an answer habit. We collect 1.1 million replies from eight models in six la...

Hao-Nan Huang, Yu-Wei Lu, Cheng-Kun Cai · 0 citations
Open access Oct 2026

Do LLM Judges Agree with Humans? Evaluating Financial Commentaries from Material Facts

The rapid adoption of large language models (LLMs) in finance has enabled automated generation of in-domain texts such as earnings summaries, market analyses, and commentary on regulated disclosures. Generating accurate and accessible financial commentary from material facts poses challenges related to domain-specific...

Gabriel Assis, Daniela Vianna, Livy Real et al. · 0 citations

Chronologic: Measuring Language Models' Ability to Represent the Past

Language models are appealing tools for research on the past. But to trust the evidence a model provides, researchers need to know whether its responses fit the period represented. Validation is challenging, because this is not a task living people ordinarily perform, and because many questions have multiple correct an...

T. Underwood, Zi-Liang Qiu, Sarah Griebel et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.