Skip to content
Book Open access

Towards Verifiable Fairness Testing: Auditable Coverage over Data-Supported Input Domains

Oct 2026 · Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering · pp. 1258-1269 · 0 citations · 34 references

TL;DR

This paper redefines black-box fairness testing as an auditable process on a data-supported, reasonable input domain and designs a structure-aware intra-unit search mechanism that combines a structure-aware discriminative field (SDF) and directional influence refinement (DIR).

Abstract

Black-box models are increasingly deployed in high-stakes domains such as education, personnel hiring, and credit assessment, making systematic and auditable fairness testing particularly important. In such settings, black-box fairness testing is typically modeled as a search problem aiming to discover biases in the target model as much as possible. However, existing methods offer limited support for verifiable evaluation on a clearly stated, data-supported test domain: which reasonable regions have been tested, which have not, and whether exploration is balanced across the test domain remain unclear. This paper redefines black-box fairness testing as an auditable process on a data-supported, reasonable input domain. We first construct a reasonable region from the reference data and divide it into a finite set of local coverage units. Then, we introduce a coverage scheduler that distributes testing work among the units and explicitly records tested and untested regions, thus achieving reproducible and verifiable sufficient testing. To improve discriminative discovery capabilities without compromising the coverage foundation, we further design a structure-aware intra-unit search mechanism that combines a structure-aware discriminative field (SDF) and directional influence refinement (DIR). Experiments on multiple real-world tabular datasets show that our method enables auditable unit-level coverage accounting and achieves more balanced coverage, higher sample naturalness, and higher discrimination discovery effectiveness than representative black-box fairness testing baselines. Moreover, the discovered instances are effective for mitigating individual fairness violations through retraining.

Read PDF

Similar papers

#machine learning Preprint Oct 2026

Tolerance-Based Fairness Auditing: Violation Certification and Sensitivity Screening

A unified tolerance-based fairness auditing framework for two complementary auditing objectives: violation certification, which prioritizes control of false violation declarations, and sensitivity screening, which prioritizes reducing missed violations is developed.

Jie Tang, Chuan-Long Xie, Li-Xing Zhu · 0 citations

Efficient Active Auditing of Multi-Group Fairness with Bias Probes

Over the past decade, Machine Learning (ML) has been trained under dual objectives: minimizing prediction error via Empirical Risk Minimization (ERM) while controlling unfairness bias. In practice, however, fairness-aware training often yields limited improvements over standard ERM, making reliable post hoc auditing es...

Ayoub Ajarra, Debabrota Basu · 0 citations
#artificial intelligence Conference Open access Sep 2026

One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs

It is argued that BBQ-style multiple-choice abstention benchmarks measure a single structural cue, and a model that solves them does not thereby become fair, and is called for evaluation suites that cover a broader spectrum of fairness alignment.

Nai-Hao Deng, Samee Arif, Shuai-Chen Chang et al. · 0 citations
#artificial intelligence Review Sep 2026

FinancialAuditBench: Benchmark Construction under Differential Privacy Using Real-World Priors

FinancialAuditBench is introduced, a benchmark for evaluating agents on financial statement audit tasks, along with a framework for systematically generating synthetic engagements for model evaluation and training in privacy-sensitive domains.

Jerry Huang, Sarvesh Babu, Matt Van Buren et al. · 0 citations
Open access Oct 2026

Elevating Test Security in Validity Theory

Cheating and other threats to score integrity have long challenged language testing, and concerns have intensified with remote proctoring and generative artificial intelligence. Yet prevailing validity frameworks treat test security primarily as an operational matter rather than as a component of validity. We argue tha...

Ramsey L. Cardwell, Daniel R. Isbell, Jieun Kim et al. · 0 citations

When Post-Processing Fairness Constraints Help and When They Harm: Evidence from Eight Cross-Domain Evaluations

FAPE (Fairness Auditing for Production Environments), a four-stage framework evaluating a single post-processing intervention, Fairlearn's ThresholdOptimizer, across eight domain evaluations: criminal justice, income prediction, legal admissions, credit lending, agricultural lending, a multi-domain benchmark corpus, he...

Nithin Raghava Ramachandra Narla · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.