Skip to content

A Critical Analysis of Empirical Research in Software Testing

2007 · International Symposium on Empirical Software Engineering and Measurement · pp. 1-8 · 89 citations · ⚡ 5 influential · 22 references
Computer Science
View source

Similar papers

Open access Aug 2026

Comparative study of PHP code static analysis tools

The purpose of the research is a comparative analysis of static PHP code analysis tools to assess their effectiveness in identifying errors, potential vulnerabilities, and typing problems in the early stages of web application development. Particular attention is paid to determining the practical applicability of the solutions under consideration in projects of various scales, as well as their impact on improving software quality, reducing the number of defects in the code and minimizing security risks. Methods . The study uses a comparative analysis method based on testing tools on a set of typical scenarios reflecting common errors and vulnerabilities of PHP applications. The tools were evaluated according to the criteria of completeness of error detection, accuracy of diagnosis, flexibility of rule configuration, ease of integration into the development process, productivity and resource consumption. Additionally, an analysis of documentation and configuration options was performed. All experiments were carried out repeatedly to ensure the statistical reliability of the results. Results . The study revealed differences in the depth of analysis, the rigor of type checking, and the mechanisms for configuring rules. It has been found that the tools exhibit varying sensitivity to logical errors, type inconsistencies, and potentially unsafe designs. Their strengths and weaknesses have been identified in the context of use in small and large projects. Conclusion . The results confirm the effectiveness of using static analysis tools as a means of improving the quality and security of PHP code. The choice of a specific solution should be based on the requirements of the project, the level of rigor of the analysis and the specifics of the development process. Regular use of such tools can significantly reduce the risk of defects and vulnerabilities, increasing the reliability and stability of web applications.

Ya. V. Ognevoy, A. Spevakov, I. Kalutskiy et al. · 0 citations
Open access Jul 2026

Software Safety Architecture Analysis

The intent of this paper publication is to address Software (SW) Safety Architecture Analysis and to provide an approach for conducting the architecture analysis Level of Rigor task utilizing software architecture patterns. While there are many architecture analysis techniques commonly used in practice today (e.g., Architecture Tradeoff Analysis Method (ATAM), Software Architecture Analysis Method (SAAM), Scenario based Architecture Level UsabiliTy Analysis (SALUTA), etc.), they seem to concentrate on the architecture of the system rather than the structure of the system’s software. By analyzing software architecture patterns, the safety practitioner can pay close attention to the true architecture of the software. The methodology described within further details the safety concerns associated with each of the patterns. The general safety concerns are stability/robustness, data handling, maintainability, testability, and security. Each of these concerns are broken down to more specificity in the brief. By identifying the software architecture pattern(s) employed in a system and understanding the safety concerns and potential mitigations, system safety practitioners have a basis for an in-depth, repeatable software architecture analysis methodology.

Myesha Dabney, Carla Adams, LaTasha Bagby et al. · 0 citations
Review Open access Aug 2026

An Empirical Analysis of Software Testing Practices and Quality Outcomes in Small and Medium Enterprises

The software industry has its own challenges of small and Medium Enterprises (SMEs) that are constrained in implementing effective testing practices, are limited in expertise and technology, and have dynamically changing technology environments. IT ecosystem with more than 35 percent of the software SMEs in India is a perfect setting to explore the correlation between testing practices and software quality deliverables. Objective: This research will focus on empirically examining the software testing criteria undertaken by SMEs in India and its effect on quality indicators, which are defect density, customer-reported defects, and time-to-market. Approach: We performed a mixed-method study that included quantitative surveys of 112 SMEs, qualitative case studies of 15 organizations, and the examination of project artifacts of Jira, GitHub, and TestRail platforms that included the period 2022-2025. Random Forest, XGBoost, and Support Vector Machines machine learning models were used to predict the quality outcomes on the basis of testing practice indicators. Findings: We have found out that automation of tests having a coverage of more than 60 percent can decrease the number of defects in production by 47 percent (p < 0.001). Those companies that have implemented CI/CD integrated testing have 34% quicker mean time to repair (MTTR). The predictive model took an accuracy of 87.3 percent in categorizing projects with regard to the quality outcome tiers. Contribution: The study offers practical information on how SMEs can maximize test expenditure, introduces a proven quality prediction model, and offers empirical evidence on the literature of software testing in emerging economies.

Priya Nandihal · 0 citations
Preprint Aug 2026

Mutation Testing of Simulink Cyber-Physical System Models: Challenges and Solutions in Practice

Several experience reports illustrate that mutation testing is capable of supporting a shift-left testing strategy, especially in industries where late bug discovery incurs very high costs and risks. In the context of cyber-physical systems, a shift left implies that engineers need to test the design models used to simulate, prototype, and analyse the feasibility of the system under design. In this paper, we analyse the challenges we encountered and the lessons we learned when incorporating mutation testing in the context of DUCO, a company producing ventilation systems for buildings. The engineers within DUCO have years of experience with model-based engineering centered around Simulink and StateFlow, including unit tests for their models. During a pilot project with our tool prototype MUT4SLX we learned that equivalent mutants, requirement traceability, and mutation testing for Stateflow represent particular challenges not yet reported in the academic literature.

Murat Kavak, Onur Kilinççeker, Serge Demeyer et al. · 0 citations
Jul 2026

Quality Assurance and Scalability: The Role of High-Test Coverage in Continuous Integration and Deployment Pipelines

The concept of productive coverage is formulated, shifting the center of gravity from gross quantitative test indicators to their semantic significance for business-critical scenarios, and practical recommendations are proposed for reconfiguring CI pipelines in order to minimize economic losses caused by brittle tests.

Evgenii Lvov · 0 citations

Related blog posts

Microsoft Research Blog Jul 30, 2026

Echoverse: Deep, evolving environments for computer-use agents

Computer-use AI agents struggle with multi-step workflows like email and customer support. Echoverse trains agents in realistic environments rather than simply providing more training tasks, helping them improve as the tasks, tests, and environments evolve. The post Echoverse: Deep, evolving environments for computer-use agents appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.