When Can We Trust Fairness Audits? Identifying Reliability Boundaries of Third-party Audit Conclusions
The Consistency Radius is introduced, a metric that quantifies the maximum distribution shift under which an audit conclusion based on a third-party dataset remains consistent, and a convex relaxation-based optimization method to estimate the radius using only model responses over the audit dataset is proposed.
Yuanhao Liu, Qi Cao, Huawei Shen
· 0 citations