Replication Package for Creating the FairMedQA Benchmark and Conducting the Related Empirical Investigation

Abstract

## 👀Overview**FairMedQA** is an adversarial medical question answering dataset for benchmarking the bias of large language models (LLMs) in the medical question answering context. FairMedQA is created from the U.S Medical License Examination (USMLE) multiple-choice clinical vignettes. Each sample includes: * An original clinical vignette from the U.S Medical License Examination (USMLE) question bank (MedQA dataset).* A neutralized clinical vignette with sensitive attributes removed* Six adversarial variants targeting: * Race (Black vs. White) * Gender (Female vs. Male) * Socioeconomic Status (Low vs. High Income) Variants are generated using a multi-agent LLM pipeline and reviewed by humans for quality control. The following figure demonstrates the workflow of the creation of the FairMedQA dataset. ## 🏠Replication Package Structure ```FairMedQA-Materials/├── 1_FairMedQA_Dataset/ │ ├── FairMedQA_Dataset.jsonl/ # Final FairMedQA Dataset based on the adversarial variants from GPT-Agent and revised by Human Reviewers│ ├── Vignette_GPT-4.o.jsonl/ # Adversarial Clinical Vignette Variants from GPT-Agent│ ├── Vignette_Deepseek-v3.jsonl/ # Adversarial Clinical Vignette Variants from Deepseek-Agent│ └── .../├── 2_Scripts/ │ ├── FairMedQA_generation/ # Python script for generating adversarial variants from neutralized clinical vignette│ ├── FairMedQA_Benchmarking/ # Python script for benchmarking given LLMs│ └── .../├── 3_Results_Raw/ │ ├── FairMedQA_Answer_{LLM_Name}.jsonl # Raw answers from {LLM_Name} on original vignettes, neutralized vignettes, and vignette variants├── 4_Results_Analysis/ │ ├── Accuracy│ └── Fairness-Heatmap │ └── results.csv #all statistical analysis result``` ## ✍️ Evaluation Metrics * **Counterfactual Fair Rate**: Consistency across counterfactual variants* **Statistical Parity Difference**: Accuracy disparity between demographic groups* **Significance Testing**: McNemar's test for evaluating answer consistency ## 📰 Details of FairMedQA Dataset **Properties:** Currently, there are 801 samples in the FairMedQA dataset. Each sample contains 39 properties, including "question id", "original question", "neutralized question", 6 "adversarial description", six "adversarial variant", 6 "variant tag", answers on original question, neutralized question, and 6 variants...