Jul 2026· Annual International Computer Software and Applications Conference· pp. 120-129· 0 citations· 22 references
Computer Science
Abstract
Software testing often depends on a test oracle to determine whether or not an output is correct. However, many real-world systems lack reliable or affordable oracles. Metamorphic testing (MT) alleviates this problem by using metamorphic relations (MRs), which examine the correctness of relations between outputs produced from related inputs across multiple executions. While the effectiveness of MT depends heavily on the quality of MRs, existing MR evaluation is typically guided by informal or study-specific criteria, limiting standardization and repeatability, especially for complex and AI-integrated systems. This paper proposes a structured MR-evaluation rubric that operationalizes key criteria to support consistent and repeatable assessment. We conducted an empirical study on nine systems under test (SUTs) spanning three levels of complexity, using fresh sets of LLM-generated MRs for each SUT. We then compared evaluations from human experts against two widely used large language models (LLMs), namely ChatGPT and Gemini, by configuring these models as MR evaluators using the same rubric. The results show that human and LLM evaluators are more consistent when assessing simpler SUTs, but their judgments increasingly diverge as system complexity grows and evaluation criteria become more varied. The study further reveals clear differences in LLM-evaluation behaviors, indicating that evaluator choice can affect MR assessment outcomes in complex settings. Overall, the proposed rubric provides a standard framework for MR evaluation, and supports a systematic comparison between human and LLM evaluators, offering empirical evidence on when LLM-based MR evaluation is more reliable, and where additional care is needed as SUT complexity increases.
An automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok is presented and a substantial gap between correctness and quality attributes is revealed.
Seyed Mohammad Mahdi Ghalandarian, Majid Bazargani, Masoumeh Taromirad· 0 citations
As Large Language Models (LLMs) are used more widely, it has become important to evaluate their fairness and prevent biased responses to sensitive attributes. A major challenge in fairness evaluation is the oracle problem, where the lack of a clear ground truth makes it hard to assess outputs with traditional testing. Metamorphic Testing (MT) addresses this by checking the consistency between source and follow-up test cases using Metamorphic Relations (MRs). However, in existing approaches, the generation of test cases is often template-driven, and the identification of sensitive attributes is manually predefined by humans, which limits coverage and relies heavily on the designer’s judgment. This study compares the quality of metamorphic test cases generated automatically by LLMs with those created manually by humans. The research involves defining Metamorphic Relations, generating follow-up cases using both LLMs and human participants, and evaluating them using expert judgment with Cohen’s Kappa to measure agreement on MR conformity and sensitive attribute relevance. By contrasting LLM-generated cases with template-driven and manually constructed approaches, this study aims to examine which method provides better coverage and alignment with fairness requirements. The results are expected to support the development of more effective and automated fairness testing for LLMs.
Francisca Isabella Manope, Adam Rafif Faqih, D. Fadhlurrohman et al.· International Conference on...· 0 citations
A large empirical study on using LLMs to generate CI configurations from natural language across services and model families suggests that similarity and validity are distinct objectives for CI generation and motivate schema-aware evaluation and tooling for LLM-based configuration generation.
This paper identifies patch verbosity as a major yet overlooked concern in LLM-based APR and proposes RECAP, a lightweight, plug-and-play adapter that attaches to existing repair frameworks after generation that achieves a substantially better size-correctness tradeoff.
Wen-Qiang Luo, J. Keung, Xiaoyu Shi et al.· 0 citations
UI component libraries are commonly assessed using execution-based metrics such as statement and branch coverage, yet these metrics provide limited insight into whether tests verify the behavioral relations implied by component APIs and documentation. This paper presents an MR-based framework that uses inferred metamorphic relations (MRs) as an empirical behavioral reference, rather than a complete specification, for assessing UI component test suites. Given a component's source, documentation, and tests, the framework infers component-specific MRs using a UI-specific taxonomy, aligns tests with the inferred relations through hybrid deterministic and semantic analysis, and computes relation-level MR coverage metrics. We manually validate both the inferred MR space and the test--MR alignment. Our evaluation shows that existing test suites exercise substantially more behavioral relations than they explicitly validate: MR Cover remains between 42.5% and 47.6% across three LLM configurations and consistently below MR Touch. Most uncovered relations are weak-oracle cases, where behaviors are exercised but lack explicit behavioral validation. MR coverage also complements execution-based coverage by revealing behavioral gaps not reflected by statement or branch coverage alone. We further assess practical relevance through issue-description mapping, oracle strengthening, and MR-relevant injected faults. Most reported issue descriptions can be mapped to inferred MR relation types; weak-oracle relations often expose missing validation evidence; and MR labels show a trend in MR-relevant fault detection. Overall, MR coverage provides a complementary relation-level perspective for assessing behavioral validation in modern UI component testing.
Yu Pei, Cunming Zhang, Jeongju Sohn et al.· 0 citations
No detectable linear relationship exists between requirement properties and oracle accuracy in this dataset, suggesting that pretraining coverage and the semantic specificity of the required behavior dominate oracle correctness.
Tiancheng Ma, Nasir U. Eisty· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
