Type-aware LLM-based Test Generation for Python Programs
TL;DR
Test4Py is presented, a novel framework that enhances type correctness in automated test generation for Python by leveraging the program’s call graph to capture richer contextual information about parameters, and introducing a behavior-based type inference mechanism that accurately infers parameter types and constructs valid test inputs.
Abstract
Automated test generation has been extensively explored, yet generating high-quality tests for Python programs remains particularly challenging. Because of Python’s dynamic typing features, existing approaches, ranging from search-based software testing (SBST) to recent LLM-driven techniques, are often prone to type errors. Hence, existing methods often generate invalid inputs and semantically inconsistent test cases, which ultimately undermine their practical effectiveness. To address these limitations, we present Test4Py, a novel framework that enhances type correctness in automated test generation for Python. Test4Py leverages the program’s call graph to capture richer contextual information about parameters, and introduces a behavior-based type inference mechanism that accurately infers parameter types and constructs valid test inputs. Beyond input construction, Test4Py integrates an iterative repair procedure that progressively refines generated test cases to improve coverage. In an evaluation on 183 real-world Python modules, Test4Py achieved an average line coverage of 83.0% and branch coverage of 70.8%, outperforming state-of-the-art tools by 7.2% and 8.4% in relative gains, respectively.