When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LLM Ordering Agents

Can LLMs Resolve Dependencies? A Benchmark for Semantic-Versioning Constraint Reasoning and Dependency Resolution

A coding agent that installs packages and untangles version conflicts is implicitly reasoning about semantic-versioning constraints and dependency resolution. Whether current language models can actually do this has not been measured, and that is the gap we address. DepResolve-Bench is a programmatically generated benchmark scored by a deterministic oracle that needs no human labels. We trust the oracle: it agrees with the production node-semver on 2250 sampled cases and, for resolution, with exhaustive search on 3000 instances. The benchmark has two tasks: deciding whether a version satisfies a range over the full npm grammar (Task A), and resolving a multi-package dependency graph or proving it unsatisfiable (Task B), with instance size as a difficulty knob. Five frontier and lightweight API models agree. They have nearly mastered the atomic rules, with Task A macro accuracy spanning 0.92 to 1.00, yet they cannot reliably put those rules together. On resolution the best model reaches 0.91 and the weakest 0.29, and every model degrades as the graphs grow, the strongest falling to 0.82 on the largest instances. A missed satisfiable instance almost always breaks a transitive constraint; on an unsatisfiable one, weaker models invent a resolution anyway (GPT-4o-mini for 44 of 80 such cases, the strongest model for one). How much we let a model reason matters more than which model it is: capped at 200 output tokens every model sits near 0.1, and accuracy climbs as the budget grows. We release the generators, oracle, and item banks; the results argue for keeping a real resolver in the loop rather than handing dependency management to a language model.

SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or>=2.0,<3, yet their grasp of version-constraint semantics has never been measured directly. We introduce SemVerBench, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo): 240 machine-checkable items with unique answers, built author-neutrally from four balanced sources (each ecosystem's official test suite plus three frontier LLM proposers) and labeled by a non-circular two-implementation oracle. Evaluating six frontier models, we find systematic, predictable per-mechanism blind spots: a partial-comparator carry rule (>1.2 means>=1.3.0) traps every model on Cargo (near 60%), and although standard PEP 440 prefix matching is universal, on zero-pad/post-release corner cases GPT-5.1 collapses (0/26) while Claude stays at 97-100% (verified on a 67-item oracle-validated set). Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar). The failures look more like an activation/application gap than a knowledge gap: injecting the rule or a light correct hint recovers most errors, whereas interval decomposition does not, and models are at ceiling on the basic forms of the same rules. An author-stratified analysis finds no statistically significant self-favoritism. Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches ~100%: coding agents should delegate version resolution to a resolver rather than reason about versions in-head.

VirtualSet: Typed Ontology Worlds as an LLM Generation Target for Grounded Queries and Guarded Decisions

Large language models increasingly read and act on enterprise data, but SQL gives a late error signal: hallucinated fields or relations can execute and return plausible wrong answers, while incorrect writes cannot be safely assessed after execution. We present VirtualSet, a live, receiver-typed ontology-world interface and generation target for LLMs. Instead of SQL, the model emits set expressions over entity-edge worlds. Generic Constraint Projection (GCP) checks expressions before execution, while future this preserves concrete receiver types through collection chains, turning invalid fields, edges, receivers, and actions into token-anchored type errors. Type-clean reads use a SQL fast path or bounded stream interpretation, with a parity oracle checking both paths over the exercised operator space. The same substrate supports guarded decisions: actions run first in a simulated world, and world-change events require external approval before actualization. On BIRD, we lift relational schemas into typed worlds and compare VirtualSet with direct SQL while holding the model, evidence, values, zero-shot setting, timeout, glossary, repair/voting, and grader constant where possible. On a frozen 1,072-question split, VirtualSet achieves 67.5% accuracy versus 63.5% for glossary-matched direct SQL with repair and voting (+4.0 points; McNemar exact p = 0.00117) using deepseek-reasoner. Full-corpus analysis finds no engine mis-computation of a type-clean expression; remaining errors arise from model semantics or gold defects. In a 30-body guard corpus, the write chain intercepts 20/20 hallucinated action bodies with zero false positives. VirtualSet thus remains competitive on SQL's home benchmark while providing pre-execution semantics for guarded decisions.

From Natural Language Policies to Executable Obligations: A Verification Harness for Dependable In-Car LLM Agents

AgentGuardUtil is presented, the authors' entry to CAR-bench Track~1, which treats the AI planer (LLM) as a fallible proposer inside a grounded verify-and-revise loop, and its core novelty is a runtime policy compiler.

SafeQL: Search-based Refinement for Safe and E � icient LLM-based Text-to-SQL

SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools

An LLM agent calling a production API cannot distinguish a query that matched nothing from a query the server did not understand. Both return HTTP 200 with a parsable body, no exception to catch and no field to branch on. We ask what predicts which one occurred, and what it does to the agent. Auditing 721,320 parameters across 2,501 independently published OpenAPI documents, we find that 7.5% declare an enumeration and 15.2% declare any machine-checkable constraint at all, while 40.1% of documents state at least one constraint in prose that their schema does not encode. Executing 219 schema-derived perturbations against live commercial endpoints from 27 vendors, reached through a single aggregation layer (Monid) that publishes a schema and returns a run identifier for every call, we find that constraint form, not vendor identity, predicts honesty: machine-checkable constraints yielded an honest error in 111 of 111 cases, prose-only constraints failed silently in 44 of 61 (p = 2e-13). Twelve models across eight families then met these endpoints on ordinary tasks. A vocabulary that the description merely exemplifies was missed by every model on 88 of 88 attempts, while vocabularies written out in full were used correctly 88 to 91% of the time. Running the full agent loop, models detected the resulting silent failure in 12% of cases, repaired it in 0%, asserted a false negative to the user in 41%, and invented a figure in 12%. Promoting the vocabulary into the schema removes the failure, from 88 of 88 to 0 of 89. The fix is one line of schema rather than a better model. Code, schemas, perturbation sets, agent transcripts and per-call run identifiers are released at https://github.com/Jasper0122/silentprobe.