Automating End-to-End Hybrid Query Processing: Benchmark, Solution, and Insights
TL;DR
A large-scale benchmark with 60\sim 90× more queries than prior work, built on 3× more databases, an automated pipeline that can execute existing methods without manual intervention, and multi-dimensional, fine-grained evaluation metrics for comprehensive assessment.
Abstract
Hybrid queries—natural language questions over structured data that require both database capabilities and LLM reasoning—have recently emerged as a prominent research topic. However, existing solutions remain overly dependent on manual workflows, and current benchmarks are limited in scale and diversity. To bridge this gap, we present (1) HyQBench \xspace, a large-scale benchmark with 60\sim 90× more queries than prior work, built on 3× more databases; (2) AutoHyQ \xspace, an automated pipeline that can execute existing methods without manual intervention; (3) multi-dimensional, fine-grained evaluation metrics for comprehensive assessment. Through extensive experiments across multiple hybrid query approaches on diverse LLM backbones, we reveal their strengths and limitations, and identify research opportunities for advancing this emerging field. Our code and data are available at https://github.com/XMUDM/HyQBench.