Validating autonomous driving systems requires diverse, regulation-compliant test scenarios. In simulation-based testing, scenarios are defined as executable scripts. Yet automatically generating such scripts from regulatory descriptions remains an open challenge, and existing approaches face fundamental trade-offs. Retrieval-assemble methods achieve reasonable compilation rates but lack scalability, whereas retrieval-based full-script generation suffers from low compilation success rates. We present Chat2Scenic, the first iterative retrieval-augmented framework to generate scenario scripts in Domain Specific Language (DSL). Specifically, Chat2Scenic provides a chatbot interface that supports interactive scenario refinement and integrates Retrieval-augmented Generation (RAG) to ground scenario generation in regulatory knowledge and DSL syntax. Furthermore, we propose an open benchmark for scenario generation comprising 123 scenarios from various regulations, including NHTSA and United Nations Vehicle Regulations, as well as other sources. Extensive evaluation with State-of-the-Art (SOTA) Large Language Models (LLMs) demonstrates that Chat2Scenic achieves 76.42% Compilation Success Rate (CSR) and 58.17% Framework Accuracy (FA), outperforming existing methods (Retrieval Assemble with 30.08% CSR, 11.03% FA and Retrieval full script generation with 16.26% CSR, 10.86% FA). To facilitate future research, we release our code as open source at https://github.com/TUM-AVS/chat2scenic.
Abstract. Web map creation remains difficult for non-specialists because it requires the combined use of geospatial data understanding, cartographic design, front-end development, and platform-specific mapping APIs. Although large language models have recently improved natural-language-driven code generation, they still perform unreliably in web mapping tasks, where failures often arise from hallucinated API usage, heterogeneous input data structures, and the gap between code plausibility and rendered map quality. This paper presents Chat2Map, an agent framework for automated web map generation from natural-language instructions. Rather than treating the task as one-shot text-to-code generation, Chat2Map formulates web map authoring as a grounded workflow that integrates skill and tool grounding, runtime schema construction for uploaded GeoJSON and JSON files, and an iterative generate-execute- diagnose-repair loop based on runtime and visual feedback. To validate the framework, we implemented a working prototype with Tianditu as a reference API setting and evaluated it on 30 web map generation tasks covering both directive-based generation and data-driven visualization. The results show that Chat2Map achieved 80% code executability and a 70% task success rate on directive-based tasks, while also maintaining a 70% task success rate on data-driven tasks where baseline models failed to produce usable outputs. These findings suggest that reliable web map generation depends not only on stronger code models, but also on how model reasoning is connected to domain knowledge, explicit data understanding, and real execution environments.
Hongping Zhang, Peilong Ma, Cong Wang et al.· The International Archives o...· 0 citations
Large language models (LLMs) have demonstrated strong potential in graphical user interface (GUI) generation, but reliable evaluation remains challenging due to uncontrolled data distributions, noisy annotations, and limited layout scenario coverage. To address this, we propose SchemaGUI, a template-based benchmark for controllable GUI generation evaluation. By synthesizing paired natural language instructions and deterministic function-call references from parameterized interface schemas, SchemaGUI can generate thousands of deterministically annotated tasks in seconds without human labeling. Based on 1,000 evaluated instances per scenario and language across six representative bilingual scenarios, we benchmark five mainstream models, including the Qwen3.5 family, Qwen3-Coder-30B, and DeepSeek-R1. Our extensive analysis reveals three key insights. First, precise geometric spatial control remains an important bottleneck; while scaling Qwen3.5 from 4B to 27B improves Schema Feasibility from 91.56% to 99.63%, the Geometry score improves more modestly (from 67.05% to 75.30%). Second, generation difficulty is highly sensitive to layout complexity, with current LLMs excelling at simple sequential arrangements but suffering severe coordinate drift in dense grids and multi-region compositions. Third, thinking mode increases token consumption while generally reducing GUI Score, particularly for smaller models.
Jiarui Dong, Yin Cai, Zhouhong Gu et al.· 0 citations
With the capability enhancement of Large Language Models (LLMs), LLM-based agent systems have demonstrated extraordinary potential in complex task reasoning. However, challenges such as context limitations and knowledge hallucinations persist in practical engineering applications. This paper focuses on the “Design of ReAct Agent Framework Fused with RAG” and proposes a reusable engineering framework. The proposed framework utilizes the ReAct structure as the core for reasoning execution, employs strictly constrained prompts to achieve reliable tool calls (Prompt as Code), integrates Retrieval-Augmented Generation (RAG) as a domain knowledge tool, and enhances system stability through multi-turn context management and error feedback self-correction mechanisms. Based on a general-purpose ReAct project and a specific school policy QA project, this paper establishes a unified experimental setup and conducts main experiments, ablation studies, and case analyses using an author-constructed QA set based on publicly available university policy documents from a specific university. Results on 50 sampled questions show a 100% tool-call success rate for the tool-enabled variants. The ablation results further indicate that context-window management and early stopping reduce the average number of reasoning steps. However, accuracy gains remain inconclusive under the current EM/F1 evaluation and require further validation. This paper provides an engineering pathway reference for designing LLM Agent systems oriented toward real-world business scenarios.
Yiquan Deng· 2026 3rd World Conference on...· 0 citations
Training API-calling large language model (LLM) agents demands massive amounts of high-quality trajectories. However, collecting such data at scale typically requires fully implemented environments with executable APIs and realistic, pre-populated backend databases, creating a major bottleneck for scalability. To overcome this, we propose an environment-free synthetic data generation approach that leverages LLMs as on-the-fly digital world models. Given only API specifications, our method generates trajectories mimicking interactions between an agent and a stateful environment. Specifically, an LLM first generates diverse tasks solvable with the provided APIs. A teacher agent then iteratively solves each task while an LLM simulator generates coherent synthetic API responses conditioned on the task context and simulation history. Finally, an LLM judge filters the trajectories to ensure the quality of the resulting dataset. We evaluate our approach on the challenging AppWorld and OfficeBench benchmarks, which include both information-retrieval and state-changing tasks. Fine-tuning models on our synthetic data yields significant performance gains, demonstrating that effective supervision for API-calling agents can be generated without any executable environment. Our results establish LLM-based API simulation as a practical, scalable solution for training agents across diverse API ecosystems.
Seanie Lee, Sanjoy Chowdhury, Chao Jiang et al.· 2 citations
Vision-language-action (VLA) models for autonomous driving jointly produce scene interpretation, language-based reasoning, and driving trajectories. Existing evaluations often use independently selected synthetic, simulated, and physical data, so measured performance gaps can be confounded by changes in scenario content rather than genuine domain sensitivity. We propose SSP (Synthetic-Simulation-Physical), an event-matched Syn2Sim2Phy evaluation framework that anchors cross-domain comparison to the same safety-critical interaction. Starting from a synthetic long-tail video, SSP builds a validated event specification that preserves road topology, participant roles, relative motion, conflict evolution, passing order, response constraints, and event phases. Platform-specific realizations are then constructed in CARLA and on a closed proving ground and are evaluated only after transfer audits confirm preservation of mandatory event properties. SSP maps heterogeneous outputs from OpenEMMA, LLaViDA, and Alpamayo-R1 into common semantic slots and a 1 s trajectory window to assess output validity, semantic accuracy, critical-interaction recognition, trajectory quality, and risk response. Across Cut-in and vulnerable-road-user crossing cases, the macro-averaged Integrated VLA Capability Scores are 0.259, 0.291, and 0.325 in the Synthetic, Simulation, and Physical domains, respectively, while the best domain varies by scenario. Alpamayo-R1, OpenEMMA, and LLaViDA obtain scores of 0.405, 0.338, and 0.131. SSP provides a reproducible scene-transfer chain and an evidence-qualified evaluation of VLA behavior without assuming that the Physical domain is universally superior.
Haojie Feng, Peizhi Zhang, Xinrui Zhang et al.· 0 citations
Model-driven and low-code platforms rely heavily on domain-specific languages (DSLs) to capture structured system intent. While large language models (LLMs) have shown impressive capabilities in generating DSLs from natural language descriptions, proprietary models such as GPT-5 remain expensive, opaque, and often brittle in practice. This paper explores whether carefully orchestrated small language models (SLMs) can achieve comparable modeling quality while offering greater transparency, affordability, and control. We introduce a modular multi-agent framework in which small models take on specialized roles-including intent classification, DSL generation, syntactic validation, consistency checking, and automated repair. The framework supports two generation pipelines: one producing DSL code validated directly against a Lark grammar, and another generating structured JSON validated through Pydantic schemas. To address the inherent limitations of small models, we incorporate a lightweight prompt expansion step that enriches minimal user inputs with relevant domain concepts before DSL generation. We evaluated the approach across four experiments involving 30 different open-source SLMs. We measured syntactic correctness, structural similarity to GPT-5 outputs, alignment with expert-written DSLs, and overall modeling quality using independent LLM-as-a-judge evaluations. The results are compelling: feedback-driven repair dramatically increased the number of models producing valid DSLs (from 9 to 23), while prompt expansion improved structural similarity by 20-40 points and boosted F1 scores against expert references across all models. Blind evaluations by LLM judges consistently preferred the outputs from the expanded and repaired pipelines. Our findings demonstrate that architectural choices-such as prompt expansion, validation-feedback loops, and role-specialized agents-matter far more than raw model scale. This work shows that high-quality DSL generation is achievable with small, transparent models, opening the door to more accessible, costeffective, and trustworthy model-driven development platforms.
J. Baber, Gabriela González-Sáez, Nicolas Hili et al.· Annual International Compute...· 0 citations