DBLifeBench is introduced, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance, and a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving is proposed.
Abstract
Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, current evaluation benchmarks remain disproportionately fixated on Text-to-SQL tasks, neglecting the holistic Database Lifecycle-from initial schema design to post-deployment maintenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, addressing the cognitive mismatch between ambiguous natural language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate balanced performance, specialized Text-to-SQL models suffer from ``catastrophic forgetting''in non-coding phases like design and maintenance. DBLifeBench serves as a foundational step toward evaluating and building true full-stack database intelligence.
DBcover is proposed, an LLM-driven database test generation framework that performs white-box, code-aware SQL test generation through contextual reasoning, and substantially outperforms existing fuzzers.
Yan-Kai Rong, Shuang Liu, Jin-Hao Dong et al.· Proceedings of the 2026 IEEE...· 0 citations
This work introduces ExpeSQL, a zero-shot, open-source–compatible, and efficient framework that combines divide-and-conquer reasoning, Best-of-N candidate selection, and self-critique with experience-guided refinement that establishes a new paradigm for deployable, self-improving Text-to-SQL systems in dynamic, real-world environments.
DataClawEval is introduced, the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios, and it comprises 100 rigorous, end-to-end tasks spanning five execution engines.
DBA-Bench is presented, a benchmark addressing four gaps between evaluation and production operations: live-environment fidelity, outcome-first evaluation, and controlled scenario reproducibility, which uses instrumented PostgreSQL environments with active workloads, persistent state, and multi-source observations.
As Large Language Models (LLMs) become foundational to next-generation Intelligent Information Systems, the bridge between natural language interfaces and structured database systems remains a critical bottleneck. While Text-to-SQL generation enables cooperative support for complex query formulation, ensuring the reliability of these generated queries at inference time is a central challenge. Conventional methods rely on coarse execution-based signals, which may limit their ability to capture the nuanced semantic alignment required for high-stakes database environments. In this work, we propose the use of Outcome Reward Models (ORMs) as a fine-grained, probabilistic feedback mechanism for test-time verification in Text-to-SQL tasks. We introduce GradeSQL, a framework for training task-specific ORMs that assign scalar utility scores to candidate SQL queries based on their semantic correctness and alignment with database schema. Our approach is evaluated on the BIRD and Spider benchmarks across multiple open-source LLM families. Experimental results demonstrate that ORM-based verification consistently outperforms traditional execution-based heuristics.
M. Tritto, G. Farano, Dario Di Palma et al.· Journal of Intelligence and...· 2 citations
A large-scale benchmark with 60\sim 90× more queries than prior work, built on 3× more databases, an automated pipeline that can execute existing methods without manual intervention, and multi-dimensional, fine-grained evaluation metrics for comprehensive assessment.
Bo Li, Chenzhan Wang, Long-Kang Lin et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
