GPS-Relational: Generative Problem Solving with Relational Databases
TL;DR
Experimental results demonstrate that GPS-Relational constructs executable reference SQL across existing benchmark and complex generated queries, and that LLM execution accuracy degrades significantly as nesting depth increases up to 15 levels.
Abstract
Large language models (LLMs) have improved natural language to SQL translation, but evaluating their outputs remains difficult when queries become complex, deeply nested, or produced at scale. Existing evaluations often rely on standard benchmarks with limited structural complexity or on model-based judgments that may be inconsistent.We present GPS-Relational, an evaluation framework that validates LLM-generated SQL by constructing an executable reference SQL query from the same natural-language intent and database schema. GPS-Relational builds this reference through grammarguided decomposition, schema mapping, semantic validation, and SQL assembly. The reference SQL is executed to produce the expected result, which is then compared with the execution result of the LLM-generated SQL, allowing equivalent SQL forms to be accepted without requiring textual or structural matching. Additionally, GPS-Relational facilitates large-scale benchmark generation by systematically synthesizing extensive datasets of natural-language and SQL pairs with controlled nesting depth and relational complexity. These generated cases enable a systematic methodology to evaluate LLM performance across varied levels of structural complexity. Experimental results demonstrate that GPS-Relational constructs executable reference SQL across existing benchmark and complex generated queries, and that LLM execution accuracy degrades significantly as nesting depth increases up to 15 levels.