TestMiner: Software Testing Analysis for GitHub Repositories

Testing AIware Systems: A Software Engineering Survey

Overall, this survey provides a structured characterization of the current state of AIware testing research and identifies key structural challenges that must be addressed to support lifecycle-aware, reliable AIware systems.

Evaluating Automated Testing on an Open-Source Web Application Using Cypress

The experimental results show that the Cypress-based end-to-end test suite has short and stable execution times, and resilient data-cy attributes significantly reduce maintenance overhead when UI changes occur.

On the risk of coding before testing: An empirical study on LLM-based test generation workflow

Large Language Models (LLMs) are increasingly used in software engineering workflows to generate both source code and test suites. This dual capability has enabled emerging development paradigms, including test-first and agentic workflows, where a single model is producing and validating implementations. However, these approaches assume that generated tests act as independent and reliable oracles - a fundamental requirement for effective software testing. In this paper, we challenge this assumption and investigate whether LLM-generated code biases the generation of subsequent tests. We introduce and empirically study the phenomenon of error propagation, where faults in generated code are systematically replicated in associated test artifacts. This leads to cases where incorrect implementations and tests are mutually consistent, masking defects rather than revealing them. We evaluate this effect across a range of programming tasks and agentic workflows, analyzing the consistency between generated code and test assertions, with particular focus on scenarios of aligned failures. Our study examines (i) whether erroneous code artifacts bias test generation, (ii) whether such bias persists under different prompting strategies, including chain-of-thought reasoning, and (iii) how errors propagate across multi-step workflows in which intermediate outputs are reused as context. The results show that error propagation is prevalent and impactful: generating tests after faulty code significantly reduces fault detection effectiveness compared to generating tests independently (14% vs. 25%). These findings highlight a fundamental limitation of current workflows, where lack of independence between generated artifacts undermines the reliability of automated testing. Furthermore, our results expose a previously underexplored threat to validity in empirical studies relying on coupled generation pipelines.

When Testing Meets Refactoring: Catalogue, Detection, and Recommendation

Automated software testing is critical for modern software development, yet test code itself requires maintenance and refactoring to prevent decay. While theoretical test-specific code smells are well-documented, the practical automation of test refactoring—both detection and recommendation—remains incipient, leading to fragmented knowledge and inadequate tooling. This paper bridges this gap by presenting a unified catalogue of test refactoring patterns derived from a mixed-method approach involving repository mining, developer surveys, and analysis of StackOverflow discussions. To operationalize this catalogue, we extend RefactoringMiner with test-aware detection capabilities and introduce PatternTracker, a tool for identifying and tracking the evolution of test patterns across version history. Leveraging this infrastructure, we analyzed 1.7 million Java projects to generate a high-confidence ground-truth dataset of test refactorings. Our results validate the prevalence of these patterns in open-source projects and establish a rigorous benchmark for evaluating future automated recommendation systems, including those driven by Large Language Models.

Automated Inline-Test Generation without Relying on Method-Level Unit Tests

TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution

Software tests and code evolve together: a code change should be followed by new or updated tests that record the new software behavior. Yet existing test generation and update benchmarks often isolate the test from the code change, and rely on static metadata that does not verify whether a test is executable or semantically tied to the code change. This makes it difficult to evaluate whether a test automation agent understands how a code change should propagate into the test suite. We introduce TestEvo-Bench, a benchmark of test and code co-evolution tasks mined from software repositories, with two tracks: in test generation, the agent shall write new tests to capture the new software behavior; in test update, the agent shall adapt failing existing tests to the changed software behavior. Each task is anchored to a real commit history and packaged with environment configuration to support execution-grounded metrics such as pass rate, coverage, and mutation score. TestEvo-Bench is also a live benchmark: each task records the timestamp of the test and code changes, and new tasks are periodically mined by our automated pipeline, so evaluation can be restricted to tasks postdating a model's training cutoff to reduce data leakage risk. The current snapshot contains 746 test generation and 509 test update tasks, curated from 59,950 candidate co-evolution records across 152 open-source Java projects. We experiment with four state-of-the-art agents that combine strong harnesses (Claude Code, Gemini CLI, and SWE-Agent) with strong foundation models (Claude Opus 4.7 and Gemini 3.1 Pro). Results show that they achieve up to 77.5% success rate on test generation and 74.6% on test update. However, success rate is materially lower on the most recent benchmark tasks and drops significantly under limited per-task cost.