Oct 2026· Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering· pp. 1258-1269· 0 citations· 34 references
TL;DR
This paper redefines black-box fairness testing as an auditable process on a data-supported, reasonable input domain and designs a structure-aware intra-unit search mechanism that combines a structure-aware discriminative field (SDF) and directional influence refinement (DIR).
Abstract
Black-box models are increasingly deployed in high-stakes domains such as education, personnel hiring, and credit assessment, making systematic and auditable fairness testing particularly important. In such settings, black-box fairness testing is typically modeled as a search problem aiming to discover biases in the target model as much as possible. However, existing methods offer limited support for verifiable evaluation on a clearly stated, data-supported test domain: which reasonable regions have been tested, which have not, and whether exploration is balanced across the test domain remain unclear. This paper redefines black-box fairness testing as an auditable process on a data-supported, reasonable input domain. We first construct a reasonable region from the reference data and divide it into a finite set of local coverage units. Then, we introduce a coverage scheduler that distributes testing work among the units and explicitly records tested and untested regions, thus achieving reproducible and verifiable sufficient testing. To improve discriminative discovery capabilities without compromising the coverage foundation, we further design a structure-aware intra-unit search mechanism that combines a structure-aware discriminative field (SDF) and directional influence refinement (DIR). Experiments on multiple real-world tabular datasets show that our method enables auditable unit-level coverage accounting and achieves more balanced coverage, higher sample naturalness, and higher discrimination discovery effectiveness than representative black-box fairness testing baselines. Moreover, the discovered instances are effective for mitigating individual fairness violations through retraining.
A unified tolerance-based fairness auditing framework for two complementary auditing objectives: violation certification, which prioritizes control of false violation declarations, and sensitivity screening, which prioritizes reducing missed violations is developed.
Jie Tang, Chuan-Long Xie, Li-Xing Zhu· 0 citations
Over the past decade, Machine Learning (ML) has been trained under dual objectives: minimizing prediction error via Empirical Risk Minimization (ERM) while controlling unfairness bias. In practice, however, fairness-aware training often yields limited improvements over standard ERM, making reliable post hoc auditing es...
It is argued that BBQ-style multiple-choice abstention benchmarks measure a single structural cue, and a model that solves them does not thereby become fair, and is called for evaluation suites that cover a broader spectrum of fairness alignment.
Nai-Hao Deng, Samee Arif, Shuai-Chen Chang et al.· Proceedings of the 2026 Conf...· 0 citations
FinancialAuditBench is introduced, a benchmark for evaluating agents on financial statement audit tasks, along with a framework for systematically generating synthetic engagements for model evaluation and training in privacy-sensitive domains.
Jerry Huang, Sarvesh Babu, Matt Van Buren et al.· arXiv.org· 0 citations
Cheating and other threats to score integrity have long challenged language testing, and concerns have intensified with remote proctoring and generative artificial intelligence. Yet prevailing validity frameworks treat test security primarily as an operational matter rather than as a component of validity. We argue tha...
Ramsey L. Cardwell, Daniel R. Isbell, Jieun Kim et al.· Language Testing· 0 citations
FAPE (Fairness Auditing for Production Environments), a four-stage framework evaluating a single post-processing intervention, Fairlearn's ThresholdOptimizer, across eight domain evaluations: criminal justice, income prediction, legal admissions, credit lending, agricultural lending, a multi-domain benchmark corpus, he...