APIPilot first derives candidate producer-consumer dependencies from OpenAPI specifications using structural heuristics and LLM-based semantic reasoning and validates them through concrete API executions before using them for test generation, outperforming both LLM-based and traditional REST API testing baselines.
Abstract
Testing RESTful APIs requires generating sequences of API calls that satisfy dependencies among operations, parameters, and runtime-created resources. Recent LLM-based approaches infer such dependencies and generate test sequences from OpenAPI specifications, but they often treat LLM-inferred relationships as correct without execution-based validation. This can introduce spurious dependencies, miss feasible operation chains, and produce infeasible tests. In this paper, we propose APIPilot}, an execution-validated framework for REST API testing. APIPilot first derives candidate producer-consumer dependencies from OpenAPI specifications using structural heuristics and LLM-based semantic reasoning. It then treats these dependencies as hypotheses and validates them through concrete API executions before using them for test generation. The validated dependencies are organized into a dependency graph from which APIPilot constructs coverage-aware workflows via bounded top-k graph traversal, separating semantic dependency inference from sequence construction. To improve subsequent tests, APIPilot further performs response-driven refinement: runtime responses are analyzed to update resource pools, adjust input-generation constraints, and prune or revise invalid dependency mappings. Empirical evaluation on 16 real-world REST API services shows that APIPilot achieves 92.3% operation coverage, up to 58.6% code coverage, and an 88.1% workflow execution success rate, outperforming both LLM-based and traditional REST API testing baselines. APIPilot also detects 197 unique 5xx failures and specification-execution mismatches, demonstrating the benefit of grounding dependency inference in execution feedback.
Context: REST APIs are widely used in industry. These APIs use HTTP for their communications. Failures in following the specifications of HTTP can lead to confusing and hard to use APIs, with possibly serious software faults with dire consequences. Objectives: Define novel automated techniques to automatically find HTTP semantics-level faults in existing REST APIs. Methods: We extended the state-of-the-art fuzzer EvoMaster with 9 new oracles to detect HTTP semanticslevel faults. Once the standard fuzzing process is finished generating N test cases, a new phase is executed in which these N tests are used as a starting point to create new scenarios (i.e., new sequences of HTTP calls) aimed at validating specific HTTP properties defined in these 9 oracles. Results: Experiments on 9 artificial APIs with inject faults show that our novel techniques can successfully detect all of them. Further experiments on 36 APIs from the WFD corpus show that our novel techniques can automatically find 166 existing faults in these real-world APIs. Conclusion: REST APIs use HTTP, and, as such, they need to follow its semantics to avoid misleading their clients and introducing subtle software faults. The novel techniques presented in this paper are shown to be effective at automatically finding several of this type of faults.
A Petri-net-guided methodology for test generation over concurrent stateful Rust APIs that represents API resources, lifecycle conditions, and causal dependencies as colored tokens and transitions; derives legal deep-state, near-legal, and partial-order concurrent scenarios; and uses them as a constrained intermediate representation for LLM-based code synthesis.
This paper addresses automated unit test generation with large language models (LLMs). LLM-based test generation has not yet attained a quality level sufficient for practical use in industry. Although LLMs often reproduce API syntax faithfully, they frequently disregard semantic usage constraints and execution-environment dependencies, leading to assertion failures, mock-related errors, and reference/resolution errors. A prior failure analysis of Java unit test generation using GPT-4o classified 2980 trials into eight failure patterns and identified three root-cause mechanisms: external context ignorance, internal context ignorance, and a syntax–semantics gap. Building on that analysis, this paper proposes a prompt design comprising three strategies: (1) making the execution state explicit in the generated test, (2) stating semantic constraints explicitly, and (3) injecting environment constraints prior to generation. In contrast to generic techniques such as few-shot learning or chain-of-thought prompting, each proposed strategy is tied to a specific root-cause mechanism, yielding a systematic design in which each rule is explicitly justified by its correspondence to a specific root-cause mechanism. Experiments on 298 methods with five models (GPT-4o, GPT-5, GPT-5.1-Codex, Claude Sonnet 4.5, and Gemini 2.5 Pro) show improved test execution success rates for every model, with absolute gains ranging from 1.1 to 21.1 percentage points (pp). Mock-related errors were reduced by 61.9%–99.2% relative to the baseline prompt, demonstrating effectiveness against the targeted failure patterns. Finally, conditions under which the strategies transfer to other code-generation tasks are discussed, along with limitations on their scope.
Software developers rely on packages to reuse existing functionality instead of implementing everything from scratch. Python developers commonly provide package and interpreter dependencies using configuration files, such as requirements.txt or setup.py. Package managers in Python, such as pip, can install packages according to dependency and interpreter version constraints specified in configuration files. However, Python dependency resolution remains challenging: (1) different packages may require incompatible versions of the same dependency; (2) dependencies may require a Python interpreter version that is incompatible with the interpreter used for the project, making a valid environment impossible; and (3) pip, the most popular Python package manager, resolves conflicts via backtracking, repeatedly trying candidate versions without knowing whether a valid execution environment exists or not. To address these challenges, we present SMTpip, an interpreter-aware environment inference technique for improving the executability of Python source-code artifacts. SMTpip constructs a dependency knowledge graph using metadata stored in the Python Package Index (PyPI) that hosts millions of package releases, encodes both package version constraints and interpreter compatibility constraints specified in configuration files into Satisfiability Modulo Theories (SMT) formulas. Solving these formulas identifies a set of package versions and an interpreter version that jointly satisfy all declared constraints. Empirical evaluation on multiple datasets from open-source Python projects shows that SMTpip achieves substantial speedups -- $6.9\times$ over pip, $9.6\times$ over Conda, $3.2\times$ over smartPip, and $4\times$ over PyEGo -- while consistently producing constraint-consistent environments.
S. Sakib, Muhammad Asaduzzaman, Curtis Bright· 0 citations
This approach uses LLMs to infer candidate specifications solely from test code and dynamic execution traces: the LLM observes only the program interface, selected inputs, and corresponding outputs or state changes, while the implementation internals remain hidden.
Tianhai Liu, Maximilian Müller, Tobias Hey et al.· 0 citations
EntailLLM is presented, which validates each LLM-proposed analyst path by entailment: the path is a traversal of the binary's function call graph, the domain knowledge is represented in a separate graph, and verification aligns the two under temporal annotated logic.
Kaustuv Mukherji, J. Patil, C. Payne et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
