TDiFf: Detecting Bugs in DataFrame Systems via Transferred DBMS Test Cases

Abstract

DataFrame systems are core components of modern data analytics platforms and achieve high performance through various optimization techniques. However, these optimizations can introduce subtle defects that are difficult to detect, undermining system reliability. Existing testing approaches primarily rely on manually written unit tests or automated unit test generation tools, which often produce invalid or low-quality test cases and have limited ability to exercise complex API sequences and optimization-induced behaviors. In this paper, we present TDiFf, a novel LLM-based testing framework for DataFrame systems. TDiFf leverages the database-like characteristics of DataFrame systems by translating SQL query statements into Python DataFrame test cases for differential testing. To address limitations of LLMs in mapping SQL constructs to DataFrame APIs and preserving semantic consistency, TDiFf incorporates feature knowledge extracted from SQL and DataFrame API documentation. In addition, it utilizes data flow graphs derived from SQL query plans to guide the generation of syntactically valid and semantically consistent DataFrame API sequences. We evaluate TDiFf on six widely used DataFrame systems: Pandas, Dask, CuDF, Modin, PySpark, and Polars. The results show that more than 99% of the generated DataFrame test cases are valid. Using these test cases, TDiFf uncovers 35 unique and previously unknown bugs confirmed by developers, and outperforms state-of-the-art baseline approaches by 2.5× to 14×.