Overall, this survey provides a structured characterization of the current state of AIware testing research and identifies key structural challenges that must be addressed to support lifecycle-aware, reliable AIware systems.
Abstract
Foundation models, particularly large language models, are increasingly embedded as core components of software systems. This shift has given rise to a growing body of research on testing such systems, referred to in this paper as AIware systems. While prior work proposes numerous techniques to expose undesirable behaviors, it remains unclear how these approaches align with established software testing practices and support the software lifecycle. This survey analyzes the AIware testing literature through the lens of classical software engineering concepts. We examine testing levels, oracle strategies, automation readiness, and diagnostic support, and assess how existing approaches map to lifecycle activities such as integration testing, regression testing, and CI-integrated workflows. Our results show that the literature is strongly concentrated on system-level, pre-release evaluation, with limited operational support for integration, regression, and deployment-time testing. We further show that many of these gaps stem from fundamental challenges in oracle design, including non-determinism, underspecified correctness, and limited diagnosability. Without stable and automatable decision criteria, AIware testing techniques remain difficult to integrate into continuous development and maintenance pipelines. Overall, this survey provides a structured characterization of the current state of AIware testing research and identifies key structural challenges that must be addressed to support lifecycle-aware, reliable AIware systems.
Microservice-based systems evolve continuously through API changes, service splits and merges, dependency churn, and deployment-topology drift driven by DevOps practices. This rapid evolution erodes the assumptions encoded in formal models, test suites, and architectural descriptions, leaving practitioners with limited guidance on how to keep verification artifacts aligned with evolving implementations. Prior surveys cover microservices design, deployment, performance, and isolated verification techniques, but, to our knowledge, none consolidate the intersection of evolution and formal/rigorous verification, nor map how repository-derived signals and CI/CD pipelines support continuous verification of microservice-based systems. This paper addresses that gap through a systematic literature review (SLR). Following the Kitchenham et al. guidelines and PRISMA-style reporting, we defined a review protocol with four research questions covering (i) formal modeling and verification approaches, (ii) the impact of architectural and API evolution on verifiability, (iii) repository-derived signals for updating formal artifacts, and (iv) continuous and incremental verification in CI/CD pipelines. Structured searches in IEEE Xplore, ACM Digital Library, SpringerLink, and Scopus returned 1187 records, which were screened in seven stages and reduced to 18 included studies (13 primary studies that supply evidence and five background studies retained for transparency and framing) through inclusion/exclusion criteria and a weighted 15-item quality-assessment instrument. We synthesized the included studies thematically and compared them along five dimensions: verification technique, evolution dimension, automation level, empirical evidence, and tool/artifact availability. The synthesis shows that actor-based model checking, TLA+ control-plane verification, static code analysis, continuous certification, and runtime/self-adaptive testing form five complementary clusters, but empirical evaluation is dominated by small academic case studies, only a minority of approaches automate model updates from repository signals, and few are integrated end-to-end into CI/CD. We discuss methodological limitations of the body of work, including a lack of industrial-scale validation and shared datasets, contrast intrinsic (build-time, model-driven) and extrinsic (recovery-based) verification strategies, and outline a research agenda toward repository-aware, pipeline-integrated verification for evolving microservices. The present synthesis distils a set of immediate, evidence-based actions for practitioners together with a focused agenda of open research challenges.
R. Gómez, Ebeid Elsayed, Enrique R. Zarate et al.· Software· 0 citations
The growing adoption of Large Language Models (LLMs) in Software Engineering has reinforced the expectation that coding activities can be largely automated. However, this perception may represent yet another historical search for a solution capable of eliminating the inherent challenges of software development. This article discusses the transition from a code-centered paradigm to Specification-Driven Development. We argue that artificial intelligence reduces some of the effort associated with writing source code, but it does not eliminate the complexity of developing professional software systems. Instead, it shifts this complexity toward domain understanding, requirements elicitation, specification development, validation, maintenance, and software evolution. Building on this perspective, we discuss the renewed centrality of Requirements Engineering, considering its implications for productivity and software quality, as well as risks associated with automation bias, ambiguity propagation, Specification Overfitting, and the accumulation of Specification Debt. Finally, we propose the Specification Paradox: the more capable artificial intelligence systems become at automatically generating software, the greater the dependence on correct, complete, verifiable, and explainable human-produced specifications. We conclude that the future of Software Engineering will depend not only on machines'ability to generate code, but also on humans'ability to correctly specify, evaluate, and evolve what is intended to be built.
Non-functional testing (NFT) encompasses critical quality assurance disciplines that determine the operational viability of large-scale software systems in production environments. This paper presents a comprehensive framework for integrating Load & Performance Testing (LPT) and Security Testing (ST) into the software development lifecycle of enterprise-grade applications. The proposed framework addresses scalability bottlenecks, throughput degradation under concurrent load, and security vulnerabilities that emerge specifically under high-traffic conditions. We examine industry-standard methodologies, tool ecosystems (Apache JMeter, Gatling, OWASP ZAP, Burp Suite), and architectural patterns for testing pipelines. A multi-phase testing workflow is introduced, covering smoke tests through soak tests and encompassing OWASP Top 10 validation and CVSS-scored vulnerability assessments. Empirical analysis across three enterprise application domains demonstrates that integrated NFT strategies reduce production incidents by 67% and security breaches by 54% compared to siloed testing approaches. The framework is designed for continuous integration and delivery pipelines, enabling automated quality gates for production deployment.
Urvish Gajjar, H. Shah· International Journal Of Eng...· 0 citations
Software systems have unique testing characteristics. Some projects can emphasize unit tests, while others may focus on end-to-end testing. Test organization may vary across ecosystems: in languages like Python and Java, tests are typically placed in dedicated folders, whereas Go and Rust projects commonly co-locate tests with source code. These distinctions make it harder to understand how a project approaches testing. In this paper, we present TestMiner, a tool for exploring software testing in GitHub repositories. TestMiner provides an overview of a project's testing practices, including test statistics, test location, test metrics across releases, and dependencies related to testing. We used TestMiner in an undergraduate Software Testing course, where 50 students explored the testing practices of real-world GitHub repositories. Overall, students expressed positive feedback regarding TestMiner. They were able to critically explore a variety of testing practices, including test organization, test evolution, test fixtures, mocking, and edge-case testing. TestMiner is available at: https://andrehora.github.io/testminer. Screencast: https://youtu.be/w1sBgLTq-7Y.
André C. Hora, José Miguel Rojas, Romain Robbes· 0 citations
The goal of this study is to understand practitioners'decision making process and security considerations in selecting and integrating AI components through an exploratory semi-structured interview study, and distill the findings into actionable recommendations for AI adopters, model providers, and researchers.
Mahzabin Tamanna, Elizabeth Lin, Sparsha Gowda et al.· 0 citations
Integrating GenAI into the software development lifecycle represents a critical advancement for current application development practices, development and maintenance. AI tools such as ChatGPT, Copilot, GitHub and Amazon CodeWhisperer tools are used by the engineers to automate routine tasks such as generation of code segments, documentation and assistance with debugging and test case creation. These functions are also to help engineer to be more efficient and to minimize the difficulty for junior level software developers to join the development. To provide ready access to information and help during development. While on the one hand it is useful but also it has its share of problems. Dependency on AI generated code is not yet considered to be secured. These are particularly used in the most critical systems where minor error may put big safety issue. There are significant concerns about intellectual property, originality of code and data privacy. Many GenAI models rely on data from openly available resources, where copyrights and sensitive information could be incorporated, as well as concern over engineers depend on AI. This can impact the fundamental problem solving capabilities of engineers and innovation will decrease. The paper is trying to deeply analyze the pros and cons of implementing GenAI into software development, analysing current applications used in software development life cycle (SDLC), drawing upon case studies and programmer experience, examining effect on code quality, team working and project timeline.This paper offers guidance in right implementation, identifying and outlining the best practice methods when integrating AI tools into software development. Examining both the opportunities and drawbacks of GenAI allows for greater understanding of how modern tools can be best utilized within the context of academic research and enterprise IT applications.
P. Arun, Jagdale, Deepti Ameta et al.· Economic Sciences· 0 citations