LLM-Based Automatic Metamorphic Test Case Generation for LLM Fairness Testing

Abstract

As Large Language Models (LLMs) are used more widely, it has become important to evaluate their fairness and prevent biased responses to sensitive attributes. A major challenge in fairness evaluation is the oracle problem, where the lack of a clear ground truth makes it hard to assess outputs with traditional testing. Metamorphic Testing (MT) addresses this by checking the consistency between source and follow-up test cases using Metamorphic Relations (MRs). However, in existing approaches, the generation of test cases is often template-driven, and the identification of sensitive attributes is manually predefined by humans, which limits coverage and relies heavily on the designer’s judgment. This study compares the quality of metamorphic test cases generated automatically by LLMs with those created manually by humans. The research involves defining Metamorphic Relations, generating follow-up cases using both LLMs and human participants, and evaluating them using expert judgment with Cohen’s Kappa to measure agreement on MR conformity and sensitive attribute relevance. By contrasting LLM-generated cases with template-driven and manually constructed approaches, this study aims to examine which method provides better coverage and alignment with fairness requirements. The results are expected to support the development of more effective and automated fairness testing for LLMs.