The results support a trade-off interpretation rather than a universal ranking: additional structural constraints changed failure modes and efficiency, but did not monotonically improve correctness or solve ambiguity and multi-turn state consistency.
Abstract
Natural-language interfaces to enterprise data must translate underspecified requests into governed, executable behavior while controlling invalid queries, policy failures, cost, and nondeterminism. SemPlan Benchmark evaluates this architectural design space with a deterministic synthetic bilingual benchmark containing 1,800 cases in English and Brazilian Portuguese; 1,200 cases form the frozen scientific evaluation subset. Four architectures are compared under the same model configuration: direct SQL generation (A1), a bounded tool-agent baseline (A2), structured semantic-request generation followed by deterministic planning and execution (A3), and a clarification/stateful semantic-plan variant (A4). Across 4,800 primary records, answer correctness was low in absolute terms: 22.25% for A1, 22.58% for A2, 25.67% for A3, and 24.25% for A4. A3 had the highest observed correctness and significantly exceeded A1, A2, and A4 in the pre-specified paired correctness analysis, while A1 retained the highest policy-correct rate and the lowest unsafe-or-invalid rate. A4 had the lowest mean API cost and lowest false-refusal rate. On a preselected 150-case stability subset, answer-correct repeatability ranged from 92.00% to 98.67%. The results support a trade-off interpretation rather than a universal ranking: additional structural constraints changed failure modes and efficiency, but did not monotonically improve correctness or solve ambiguity and multi-turn state consistency.
This semantic path compilation (SPC) system against direct DDL-to-SQL generation on the ACME insurance benchmark supports an end-to-end systems result, not a causal claim that compilation alone produced the gain, because SPC receives governed semantic artifacts that the DDL baseline does not.
This work presents an agentic text-to-SPARQL system that goes one step beyond static tool-using agents: a researcher agent that, after each round of inference on a validation set, proposes and tests changes to its own prompts, rules, and tool-orchestration code.
Natural-language-to-SQL systems have ad- vanced rapidly on academic benchmarks, yet production enterprise schemas exhibit graph- like, semi-structured, deeply nested structure that current benchmarks do not measure. We make two complementary contributions. First, we introduce the DevRev NL2SQL bench- mark: 900 execution-verified queries with nested-type and link-graph structure, accom- panied by the Semantic Depth Score (SDS), a schema-agnostic rubric for analytical reasoning depth. Second, we present a cost-aware single- generation agentic architecture whose schema- selection, metadata-retrieval, and error-repair components are designed for the requirements this regime imposes. On the DevRev NL2SQL benchmark the system attains 91.7% answer correctness, a margin of 54.6 percentage points over the next-best baseline; on the Spider 2.0 Snowflake public dataset, it is competitive with leading systems at a single-generation operating point.
Yoga Sri Varshan Varadharajan, Ajay Yadav, Ritesh Goru et al.· 0 citations
A coding agent that installs packages and untangles version conflicts is implicitly reasoning about semantic-versioning constraints and dependency resolution. Whether current language models can actually do this has not been measured, and that is the gap we address. DepResolve-Bench is a programmatically generated benchmark scored by a deterministic oracle that needs no human labels. We trust the oracle: it agrees with the production node-semver on 2250 sampled cases and, for resolution, with exhaustive search on 3000 instances. The benchmark has two tasks: deciding whether a version satisfies a range over the full npm grammar (Task A), and resolving a multi-package dependency graph or proving it unsatisfiable (Task B), with instance size as a difficulty knob. Five frontier and lightweight API models agree. They have nearly mastered the atomic rules, with Task A macro accuracy spanning 0.92 to 1.00, yet they cannot reliably put those rules together. On resolution the best model reaches 0.91 and the weakest 0.29, and every model degrades as the graphs grow, the strongest falling to 0.82 on the largest instances. A missed satisfiable instance almost always breaks a transitive constraint; on an unsatisfiable one, weaker models invent a resolution anyway (GPT-4o-mini for 44 of 80 such cases, the strongest model for one). How much we let a model reason matters more than which model it is: capped at 200 output tokens every model sits near 0.1, and accuracy climbs as the budget grows. We release the generators, oracle, and item banks; the results argue for keeping a real resolver in the loop rather than handing dependency management to a language model.
Zhuo-Xi Wang, Haoyu Zhang, Jing-Wen Hou et al.· 2026 8th International Confe...· 0 citations
Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standard evaluation criteria such as syntactic validity or planner success can substantially overestimate faithfulness to the described task: a generated problem may be parseable and solvable while misrepresenting the intended initial state, goal, object structure, or optimization target. This paper studies an end-to-end NL-to-PDDL pipeline that combines LLM generation, checks in terms of PDDL parsing, planning and validation, a domain-conformance checker, an LLM critic, and iterative repair. Fine-grained repair feedback is constructed from the domain description, the generated problem, the natural language problem description, and operational diagnostics. Reference-based comparisons against curated benchmark PDDL problem descriptions are used for post-hoc benchmark analysis, and these offline checks include renaming-invariant structural matching and semantic equivalence, where domain support is available. Across Planetarium, AutoPlanBench, and curated PDDL~2.1 problems, results show that operational success and benchmark-reference reconstruction can diverge substantially. Results also show that structured repair can be useful, and that PDDL~2.1 remains challenging for reference reconstruction, even when operational success improves.
J. Rosa, Pedro Santos, Valdemar Oliveira et al.· 0 citations
guided table retrieval is presented, a four-phase pipeline that combines deterministic grounding via hash-based predictors, structural exploration of join-graph reachability, LLM-powered disambiguation of sources and targets, and algorithmic merging into minimal, topologically ordered join trees.
Alekh Jindal, J. Pandey, C. Pavlopoulou et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
