This work mathematically demonstrates that standard sampling methods fail efficiently for rare violations, motivating a direct structural approach and introduces an exact sampling procedure and closed-form risk statistics that provide feature-level attributions without requiring repetitive random trials or complex sampling algorithms.
Abstract
Data-driven decision pipelines combining predictive machine learning models with downstream optimization software are increasingly used to make high-stakes operational decisions. Certifying the safety, fairness, and reliability of these decisions is essential, yet traditional scenario generation methods rely on repeated random testing, which becomes computationally prohibitive when failure events are rare and offers little insight into why failures occur. We present a framework for local violation certification designed specifically for linear decision pipelines under input uncertainty. We mathematically demonstrate that standard sampling methods fail efficiently for rare violations, motivating a direct structural approach. By analyzing the fixed decision boundary of a deployed pipeline, we show that the local risk of failure can be calculated directly in closed form using a single optimization solve. Furthermore, we introduce an exact sampling procedure and closed-form risk statistics that provide feature-level attributions (identifying which input characteristics contribute most to potential non-compliance) without requiring repetitive random trials or complex sampling algorithms. We demonstrate our approach on an economic power dispatch system subject to emissions regulations, delivering precise, auditable risk assessments at a fraction of the traditional computational cost.
PluginEval is introduced, a benchmark constructed through a two-stage framework that systematically mitigates limitations of Large Language Models and evaluates five model families, including proprietary models and models with open weights, analyze their performance across difficulty levels and error categories, and validate the judge through agreement with human annotations.
Accurate Online Failure Prediction (OFP) has been shown to be feasible in Operating Systems (OSs) settings, but prediction alone is not sufficient for practical adoption. Without diagnostic insight, operators have limited basis to trust alerts or decide how to respond. Moreover, even when predictive accuracy is high, it is often unclear whether models are capturing meaningful failure processes or merely exploiting workload-specific noise and incidental correlations in telemetry. This paper reports a practical experience building and evaluating an explainable OFP pipeline for Linux OSs. We combine consensus-based feature selection for detection with temporal onset analysis, subsystemlevel causal analysis, and complementary diagnostic mechanisms to support failure interpretation. Evaluated under strict crossworkload conditions with frozen training artifacts, it achieved 91-94% detection on unseen workloads without retraining, while maintaining false alarm rates below 1%. However, failure mode diagnosis proved substantially more sensitive to workload shift, and several diagnostics mechanisms showed limited effectiveness for specific failure types. Our experience highlights three main lessons: i) detection generalizes more robustly than diagnosis across workload changes; ii) early-warning capability depends strongly on the failure mode, ranging from 38 to 215 seconds in our study; and iii) unseen failure modes are not reliably diagnosable from related training modes alone, providing 0% accuracy under Leave-One-Mode-Out (LOMO) evaluation. Taken together, these results show the value of complementary explainability mechanisms for interpreting accurate failure predictions, revealing when predictive signals reflect transferable failure structure and when diagnostic generalization breaks down under workload variation.
Stopping-rule experiments ask participants to move through ordered stages until they decide to act. These designs are useful for studying intervention thresholds, but raw survey exports are usually stored in wide form, while correct modelling requires scenario-level thresholds and stage-level at-risk rows. This paper presents a runnable R artifact for reconstructing and validating stopping-rule data. The artifact uses a JSON configuration layer, modular R source files, toy data, a second structurally different toy configuration, automated testthat tests, adversarial validation checks, reference outputs, licensing, and citation metadata. In a de-identified sequential risk-decision case study, the same reconstruction logic maps 1,108 potential scenario observations from 277 consenting participants, flags 25 ambiguous response sequences, retains 1,083 clean scenario-level observations, and generates 3,047 stage-level at-risk rows. The evaluation documents local execution, automated test coverage, end-to-end preprocessing behaviour, a second configuration demonstration, adversarial failure behaviour, and expected linear scaling for larger survey exports. The contribution is a reproducible data-engineering system that bridges raw stopping-rule survey exports and standard event-history or repeated-measures modelling tools, while making exclusion decisions auditable rather than hidden in manual recoding.
Unknown authors· International Journal of Sec...· 0 citations
RiskBlend is proposed, a classifier-agnostic prioritization framework that combines four complementary risk signals: historical failure patterns, prediction shift, decision-boundary shift, and neighborhood change that achieves the highest average APFD in all 80 dataset-classifier-scenario combinations.
Safety classifiers deployed with large language models often fail for two reasons: their decisions reflect the policy learned during training rather than the deployer's desired policy, and their performance degrades as deployment traffic evolves. We present Regime-Conditional Verification (RCV), a lightweight wrapper that adapts an off-the-shelf safety classifier without retraining it. RCV estimates, from the classifier's internal representations, the probability that each prediction disagrees with the deployer's policy, and selectively corrects predictions likely to be wrong. The same correctness estimates also provide a label-free signal for detecting distribution shift, enabling a maintenance loop that updates the correctness estimation layer and resorts to classifier fine-tuning only when necessary. Across three off-the-shelf safety classifiers and two benchmark datasets, RCV improves adherence to the deployer's policy in every classifier-dataset combination, catching up to 0.81 of previously missed unsafe content without modifying the underlying classifier. In a deployment study with ten attack campaigns, each a harm category held out of RCV's training, RCV detects every campaign in a dedicated injection panel; in the maintenance census most drift episodes are repaired without updating the classifier, and the fine-tune is reserved for the residual episodes that repair does not restore.
