Skip to content

ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification

Sep 2026 · 0 citations · 22 references
Computer Science

TL;DR

Experiments show that ProbPlug provides more reliable confidence estimates, improves classification performance with negligible additional overhead, and exhibits strong generalization across tasks, indicating that ProbPlug serves as a practical solution for confidence estimation in LLM-based classification.

Abstract

Large language models (LLMs) have achieved strong performance across a broad range of classification settings, yet the reliability of their predictions remains a major obstacle to deployment in high-stakes scenarios. Although confidence estimation for LLMs has been widely studied, confidence calibration for LLM-based classification remains underexplored. We introduce ProbPlug, a lightweight confidence estimation framework for LLM-based binary classification, which predicts whether an output is correct using internal token features extracted from a frozen LLM. ProbPlug employs a self-attention module to aggregate hidden representations and can be integrated into the original inference pipeline without modifying the base model. Experiments across multiple tasks involving both text-based and multimodal large models show that ProbPlug provides more reliable confidence estimates, improves classification performance with negligible additional overhead, and exhibits strong generalization across tasks. These results indicate that ProbPlug serves as a practical solution for confidence estimation in LLM-based classification. Our code is publicly available at Github.

View source

Similar papers

Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models

Pinocchio is introduced, an external calibrator that estimates the correctness of responses from black-box API models that needs only a single forward pass to generate an uncertainty estimate and requires no access to the target model's logits, weights, or internal states.

K. D. Hayes, Arka Pal, Hao-Song Zhang et al. · 1 citation
#artificial intelligence Conference Open access Sep 2026

Noisy Test-Time Reinforcement Learning for Code LLMs

The Noisy Test-time Reinforcement Learning framework (NTRL-Code) is proposed, which enables robust self-evolution of code LLMs using only unlabeled noisy data during the testing stage, and employs an abstract-syntax-tree (AST)-based structural aggregation mechanism to estimate a proxy target from multiple candidate pro...

Xi-Kai Yang, Hieu Trung Nguyen, Dun-Yuan Xu et al. · 0 citations
Preprint Oct 2026

Circuits of LLM Classifiers

This work introduces circuits that organize LLM classifiers into an adaptive sequence of queries and decisions that characterize the optimal circuit, obtain the fundamental limits of achievable accuracy, and provide practical insights on realizing the optimal circuit.

Hadi Khalaf, Sajani Vithana, Kunal Talreja et al. · 0 citations
Book Open access Oct 2026

Harnessing Uncertainty in Code Language Models: Lessons from Vulnerability Detection

Code Language Models (CLMs) are increasingly adopted for vulnerability detection, yet knowing when to trust their predictions remains an open challenge. Uncertainty estimation has improved reliability in traditional discriminative models, but whether these benefits transfer to the generative paradigm of CLMs is unclear...

Hao-Dong Li, Xiao Cheng, Xu-Dong Wang et al. · 0 citations
#natural language process... Open access Sep 2026

Calibration as a First-Class Criterion in LLM Evaluation

It is argued that each NLP subfield should pair its main performance metric with a calibration score and call for treating calibration as an essential property of every model rather than a niche topic.

Mario Sanz-Guerrero, Katharina von der Wense · 1 citation
#machine learning Preprint Oct 2026

How Low Can You Go: Fine-Tuning LLMs with Bayesian Optimization (LlMBO)

Recently, Morris et al. introduced TinyLoRA, a strategy for LLM fine-tuning which introduces arbitrarily few parameters, and demonstrated accuracy improvements using less than $10$ parameters rather than the ${\gtrsim 10,000}$ typically demanded by LoRA-based methods. However, that study continued to use gradient-based...

Aditya Kulkarni, Davide Sartor, Carlos J. Soto et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.