Jul 2026· ACM Transactions on Software Engineering and Methodology· 0 citations· 43 references
TL;DR
ExReg, a human-in-the-loop workflow that automatically generates discriminative examples using SMT-based constraint solving, highlights how automated example generation guided by formal methods and mutations can improve the reliability, efficiency, and trustworthiness of LLM-assisted regex pattern generation.
Abstract
Regular expressions (regexes) are widely used in software development but remain difficult to author and validate due to their compact syntax and subtle semantics. While large language models (LLMs) can now generate regexes from natural-language descriptions, their outputs often miss developer intent, and existing refinement techniques rely on developers to craft positive and negative examples—especially discriminatory ones that expose fine-grained semantic differences. Producing such examples is cognitively demanding and often leaves regexes under-validated. This paper introduces ExReg, a human-in-the-loop workflow that shifts this burden away from developers. Given an ambiguous natural-language specification, an LLM first proposes multiple plausible regex candidates. Instead of requiring developers to devise discriminative examples, the system automatically generates them using SMT-based constraint solving. Developers need only affirm whether these synthesized test strings match their intent. The system further mutates candidates and reuses discriminative strings to systematically eliminate incorrect patterns. Across six benchmark datasets and six state-of-the-art LLMs, ExReg accurately identifies valid regexes—or determines that none are appropriate—while requiring only minimal developer validation. In particular, ExReg achieves an average accuracy of 87%, while requiring users to inspect only 5 examples on average, each with a mean length of 8 characters and an inter-example waiting time of under 13 seconds. These results highlight how automated example generation guided by formal methods and mutations can improve the reliability, efficiency, and trustworthiness of LLM-assisted regex pattern generation.
Most current tasks remain suitable for evaluating the fitting and recombination of previously unseen expressions, but are insufficient on their own to identify contributions from priors beyond a fixed search space.
Large Language Models (LLMs) have achieved impressive performance in translating natural language queries into executable SQL. However, these systems remain prone to
deceptive failures
: generating syntactically valid SQL that executes but fails to capture the user's intent. In this work, we argue that further progress can be made by focusing specifically on
verification.
In this paper, we formalize Text-to-SQL verification as a standalone task: assessing whether a candidate SQL query semantically satisfies a natural language request, without access to ground-truth labels. We propose and compare two modular verification strategies:
Round-Trip Critique
, which reverse-translates SQL into natural language to detect semantic drift, and
Synthetic Execution Consistency
, which uses unit-test-like synthetic inputs to ground verification in execution results. Our evaluation shows that these methods provide a robust signal for identifying incorrect queries, successfully flagging 64% of errors from a state-of-the-art generator and outperforming standard error detectors. We demonstrate two critical applications: (1) auditing foundational benchmarks (Spider, BIRD and KaggleDBQA), revealing that over two-thirds of "generator failures" actually stem from flawed benchmark labels, and (2) enabling Selective Generation, where a system uses verification signals to abstain from answering when confidence is low. Our results show that this paradigm significantly improves the quality of deployed data interfaces by transforming silent failures into explicit abstentions that alert the user to the failure and give them a chance to correct it.
Unknown authors· Proceedings of the VLDB Endo...· 0 citations
In fast-evolving software systems, effective 'natural language requirements parsing' and downstream change effect analysis capability across a multitude of codes represents low-hanging-fruit in this regard. We present a structured framework to deploy Large Language Models (LLMs) for automating two essential software engineering tasks, namely requirement interpretation and change impact analysis Utilizing the inherent understanding of semantics offered by transformer-based LLMs, the novel approach advances by converting vague and unstructured requirement documents into structured but machine-readable specifications to offer a direct traceability mapping from requirements to system components. Additionally, the framework leverages LLM-driven dependency analysis to predict and quantify how change effects percolate through connected modules which can minimize manual effort and human errors. This approach combines prompt engineering and retrieval-augmented generation (RAG) for domain-relevant accuracy plus fine-tuning techniques. On open-source and enterprise-grade software projects, experimental evaluations show that disambiguation accuracy, traceability precision, and change impact coverage of our approach are orders of magnitude better than state-of-the-art rule-based or static analysis tools. Notes: The results illustrate the application of LLMs at scale and demonstrate how these can alter software engineering workflows by removing bottlenecks (at a massive scale) at different stages of the software development lifecycle. In this research, we provide a generalizable pipeline that helps to bridge the gap from NLP advancements into practice for software lifecycle management.
Nithya Krishnan, Kumaran Ramanujam, Suresh Babu Narra et al.· 2026 International Conferenc...· 0 citations
Experimental evaluation on 300 realistic pattern mining tasks demonstrates consistent improvements in algorithm configuration accuracy, parameter compliance, and dataset specification correctness across zero-shot, one-shot, and few-shot settings, highlighting the effectiveness of inference-time domain grounding for enabling more reliable and reproducible pattern mining workflows without requiring model retraining.
Madhavi Palla, Uday Kiran Rage, Arjun Chakravarthi Pogaku· International Journal of Dat...· 0 citations
This analysis covers 2,857 report-test-patch triplets from Defects4J and SWT-Bench using two widely adopted instruction-tuned LLMs from distinct model families and finds that both models exhibit systematic optimism relative to humans and only modest rank agreement, motivating bias-aware evaluation.
Wendkûuni C. Ouédraogo, Yinghua Li, Xueqi Dang et al.· arXiv.org· 0 citations
Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standard evaluation criteria such as syntactic validity or planner success can substantially overestimate faithfulness to the described task: a generated problem may be parseable and solvable while misrepresenting the intended initial state, goal, object structure, or optimization target. This paper studies an end-to-end NL-to-PDDL pipeline that combines LLM generation, checks in terms of PDDL parsing, planning and validation, a domain-conformance checker, an LLM critic, and iterative repair. Fine-grained repair feedback is constructed from the domain description, the generated problem, the natural language problem description, and operational diagnostics. Reference-based comparisons against curated benchmark PDDL problem descriptions are used for post-hoc benchmark analysis, and these offline checks include renaming-invariant structural matching and semantic equivalence, where domain support is available. Across Planetarium, AutoPlanBench, and curated PDDL~2.1 problems, results show that operational success and benchmark-reference reconstruction can diverge substantially. Results also show that structured repair can be useful, and that PDDL~2.1 remains challenging for reference reconstruction, even when operational success improves.
J. Rosa, Pedro Santos, Valdemar Oliveira et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
