AutoTestAI: Bridging Solidity Source Code and Structured Test Specifications for AI-Assisted Smart Contract Validation

Abstract

Smart contract failures can cause irreversible financial and operational losses, yet current validation workflows still rely heavily on warnings, execution traces, and manually authored tests that are difficult to scale for contract-specific logic. Existing automated tools—static analyzers, symbolic executors, and fuzzers—primarily produce issue-centered outputs such as warnings or counterexamples rather than structured, reviewable test artifacts. This paper investigates whether large language models (LLMs) can bridge this gap by generating structured draft test specifications from Solidity source code. We propose AutoTestAI, an exploratory implementation-oriented framework that combines contract preprocessing, a structured auditor-style prompt, and multilayer output purification to produce machine-readable CSV test specifications suitable for downstream review and possible test implementation. On a 27-contract main benchmark, AutoTestAI achieves 91.3% overall function-level coverage. Under the revised baseline-and-ablation framework, CSV purification improves mean contract-level coverage on the Main27 split from 0.00% (strong_raw) to 91.53% (strong_purified), while enabling preprocessing within the full pipeline improves Holdout40 mean coverage from 69.73% (autotestai_no_preprocess) to 83.46% (autotestai_full), with both coverage gains statistically significant under paired Wilcoxon testing ( $p{\lt }0.01$ ). An expanded Foundry executability-convertibility validation shows 141/141 converted tests compiled and 133/141 executed successfully. A fresh 40-contract multi-project holdout set and a model sensitivity check with gpt-4o further examine broader applicability and model dependence. The method is intended as a practical complementary layer alongside static analysis, symbolic execution, fuzzing, and expert review, rather than a production-ready auditing framework or a substitute for formal verification, exhaustive testing, or expert judgment.