This work proposes the first completely probabilistic architecture-agnostic guardrail to leverage the LLM early output distributional signals for estimating and calibrating the safety probability, thereby enabling early stopping of unsafe ongoing outputs.
Abstract
Recent research on Large Language Model (LLM) safety has widely adopted guardrails to identify unsafe LLM outputs. Existing guardrails typically formulate safety assessment as a deterministic classification task, mapping a discrete token sequence to a discrete safety label. However, this paradigm has two limitations: First, safety assessment is inherently an uncertain problem, particularly during the early generation state. Second, relying solely on discrete token sequences discards the rich probabilistic information embedded in the LLM output distribution. To address these limitations, we propose the first completely probabilistic architecture-agnostic guardrail \textsc{ProbGuard} to leverage the LLM early output distributional signals for estimating and calibrating the safety probability, thereby enabling early stopping of unsafe ongoing outputs. Specifically, given an LLM's generated prefix distribution, we formulate the safety risk as the unsafe probability of its continued generation dynamics and estimate this risk by Monte-Carlo sampling. Through post-training on the distributional signals and calibrated safety risk, \textsc{ProbGuard} achieves the best calibration performance across all nine model--dataset combination settings, reducing the average Brier score and ECE by 79.6\% and 71.9\%, respectively, over the best baseline. \textsc{ProbGuard} further limits the attack success rate to at most 1\% across six representative jailbreak attacks after observing the LLM early output distributions from only the first ten decoding steps.
Safety classifiers deployed with large language models often fail for two reasons: their decisions reflect the policy learned during training rather than the deployer's desired policy, and their performance degrades as deployment traffic evolves. We present Regime-Conditional Verification (RCV), a lightweight wrapper that adapts an off-the-shelf safety classifier without retraining it. RCV estimates, from the classifier's internal representations, the probability that each prediction disagrees with the deployer's policy, and selectively corrects predictions likely to be wrong. The same correctness estimates also provide a label-free signal for detecting distribution shift, enabling a maintenance loop that updates the correctness estimation layer and resorts to classifier fine-tuning only when necessary. Across three off-the-shelf safety classifiers and two benchmark datasets, RCV improves adherence to the deployer's policy in every classifier-dataset combination, catching up to 0.81 of previously missed unsafe content without modifying the underlying classifier. In a deployment study with ten attack campaigns, each a harm category held out of RCV's training, RCV detects every campaign in a dedicated injection panel; in the maintenance census most drift episodes are repaired without updating the classifier, and the fine-tune is reserved for the residual episodes that repair does not restore.
This work introduces Gradient Activation Adaptive Multi-Level Splitting (GA-AMLS), which adapts rare-event Monte Carlo methods to the continuous activation space of language models and establishes activation space as a tractable domain for rare-event estimation in language models, circumventing the brittleness of discrete input-space search.
Nikita Y. Parulekar, Anqi Liu· arXiv.org· 0 citations
It is shown that policies within a bounded $\chi^2$ divergence from the proxy-feasible reference distribution admit an $N$-independent safety-hacking bound, and instantiate this general coverage-control principle with constrained pessimistic sampling.
This case shows how evaluation-design validity can be checked structurally before model inference and why base correctness does not determine intervention-response fidelity.
LongGuard is presented, a framework that evaluates, mechanistically analyzes, and mitigates long-context guardrail failure, and proposes two training-free mitigations - Chunked Detection and Attention-Head Sharpening (AHS) - and a deployment protocol that selects configurations by context length and audit side.
Large language model (LLM) safety judges from different model families can disagree by an order of magnitude on the same responses, with reported attack success rate (ASR) ranging from 0.04 to 0.42 on a single benign-only fine-tuned model. This is not ordinary classifier noise: the gap is driven by judges whose unsafe recall is too low to detect harmful outputs. Existing pipelines select judges by aggregate metrics such as balanced accuracy, which can look acceptable while false negatives silently suppress ASR. We propose a confidence-interval (CI) filtered two-stage judge-screening framework. Stage 1 retains only judges whose Clopper–Pearson 95% lower bound on unsafe recall exceeds a predefined floor; Stage 2 selects the highest balanced-accuracy judge among the survivors. When no judge has sufficient calibration evidence, the framework emits an explicit low-confidence or abort verdict, and downstream ASR can be accompanied by a Rogan–Gladen first-order prevalence correction. We evaluate the framework on seven labeled prompt sources and nine candidate judges spanning five base model families in two judge categories. In small-calibration regimes, it reduces the rate of selecting judges that fail the unsafe-recall screen by an order of magnitude. Across five English sources it produces a pass and reject map that is stable under the unweighted bound, with one of 45 judge-source cells reversing under design weighting; on a Chinese multilingual jailbreak source, every English-trained judge fails Stage 1 and the framework abstains. A controlled supervised fine-tuning (SFT) case study with varying benign-to-safety data ratios shows that on the same benign-only (1:0) responses the reported ASR spans a factor of 12.5 across judges, and that the lowest-recall judge underestimates the human-labeled ASR by a factor of 6.0; CI-filtered screening prevents the corresponding ASR underestimation.
Ji-Xiang Yang, Jun-Fei Yi, Jin-Han Li et al.· IEEE Access· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
