Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· 0 citations· 14 references
TL;DR
A large-scale benchmark with 60\sim 90× more queries than prior work, built on 3× more databases, an automated pipeline that can execute existing methods without manual intervention, and multi-dimensional, fine-grained evaluation metrics for comprehensive assessment.
Abstract
Hybrid queries—natural language questions over structured data that require both database capabilities and LLM reasoning—have recently emerged as a prominent research topic. However, existing solutions remain overly dependent on manual workflows, and current benchmarks are limited in scale and diversity. To bridge this gap, we present (1) HyQBench \xspace, a large-scale benchmark with 60\sim 90× more queries than prior work, built on 3× more databases; (2) AutoHyQ \xspace, an automated pipeline that can execute existing methods without manual intervention; (3) multi-dimensional, fine-grained evaluation metrics for comprehensive assessment. Through extensive experiments across multiple hybrid query approaches on diverse LLM backbones, we reveal their strengths and limitations, and identify research opportunities for advancing this emerging field. Our code and data are available at https://github.com/XMUDM/HyQBench.
This work introduces ExpeSQL, a zero-shot, open-source–compatible, and efficient framework that combines divide-and-conquer reasoning, Best-of-N candidate selection, and self-critique with experience-guided refinement that establishes a new paradigm for deployable, self-improving Text-to-SQL systems in dynamic, real-wo...
GQLBench is a new benchmark built through an automated and scalable framework that integrates NL2SQL-to-NL2GQL conversion with graph-native data generation, and enables fine-grained evaluation across dialects, graph patterns, and query complexity.
Yanning Su, Yuhang Zhou, Yang Fang et al.· Annual Meeting of the Associ...· 1 citation
Eagle, a comprehensive benchmarking framework tailored for evaluating OpsLLMs, delivers a deployable foundation for advancing large-model applications in AIOps and open-source the framework and dataset to foster community adoption and reproducibility.
Yuhe Liu, Changhua Pei, Hang Wang et al.· SIGSOFT FSE Companion· 0 citations
The proliferation of IoT and real-time monitoring has established Time-Series Databases (TSDBs) like InfluxDB as critical infrastructure. However, their functional query languages (e.g., Flux) present a steep learning curve, hindering data accessibility for non-experts. While Natural Language Interfaces (NLIs) offer a...
Xuefeng Wu, Yuanfeng Song, Jiawei Wen et al.· Proceedings of the 32nd ACM...· 0 citations
As Large Language Models (LLMs) become foundational to next-generation Intelligent Information Systems, the bridge between natural language interfaces and structured database systems remains a critical bottleneck. While Text-to-SQL generation enables cooperative support for complex query formulation, ensuring the rel...
M. Tritto, G. Farano, Dario Di Palma et al.· Journal of Intelligence and...· 2 citations
This work proposes MARS, a scalable knowledge graph question answering (KGQA) approach that requires no model fine-tuning, and performs a structured retrieval procedure that links question entities to the KG and iteratively retrieves relevant next-hop information.
Nikit Srivastava, Daniel Vollmers, René Speck et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
