A language-model-based workflow that generates, validates, and iteratively corrects ABB RAPID robot programs from natural language task descriptions is presented, showing how RAG and MCP can connect grounded code generation with executable feedback from industrial robot simulation software, while reducing but not eliminating expert setup and final supervision.
Abstract
Flexible manufacturing requires industrial robots to be reprogrammed rapidly as product variants change. This paper presents a language-model-based workflow that generates, validates, and iteratively corrects ABB RAPID robot programs from natural language task descriptions. A dual-stream retrieval-augmented generation (RAG) pipeline grounds code generation in verified technical documentation and production templates, reducing domain-specific errors produced by ungrounded language models. A custom Model Context Protocol (MCP) server connects the language-model client directly to ABB RobotStudio for automated code upload, simulation execution, and diagnostic feedback. The evaluation combines a 30-query retrieval benchmark, scoped code-generation checks, and RobotStudio case studies in a simulated pickand- place manufacturing cell. The simulation loop exposes execution failures that static and semantic checks alone cannot catch, including suction release-height errors, unreachable placement targets, and configuration-dependent recovery motions. The results show how RAG and MCP can connect grounded code generation with executable feedback from industrial robot simulation software, while reducing but not eliminating expert setup and final supervision.
Training and validation of Embodied AI for social navigation critically depends on realistic simulation environments, yet many current approaches fail to find a balance between realism and simulability. We propose D3D-GEN, a novel world generation system that combines a domain agent with a retrieval-augmented generation (RAG) pipeline grounded in that domain. Our system enables users to rapidly generate domain-grounded, fully interactive 3D worlds by automating both the collection of domain knowledge and the synthesis of realistic floorplans and object placements, without dependence on any fixed 3D model database. Given a domain description prompt, the research agent collects publicly accessible domain-specific data and constructs a persistent domain database. Using this database, our RAG pipeline generates plausible floorplans and object placements by dynamically querying a user-provided semantic database, which can be easily extended or modified. The output is a fully interactive 3D world loadable by the popular simulators Isaac Sim and Gazebo. With our approach, we have built databases for several common domains (indoor residential, hospital, office) and generated dozens of distinct, plausible simulation environments for each domain. We present D3D-GEN with a local web frontend that facilitates rapid, interactive world generation for robot simulation.
A. Do, V. Shcherbyna, Tai Duc Nguyen et al.· 0 citations
A large language model (LLM) coding agent can build complete MODFLOW 6 input file sets from natural-language descriptions when given the right reference material, without retraining the model. The reference material, which we call a plugin, comprises 42 plain-text package files (one per MODFLOW 6 package, written from the official Input/Output Guide, the Fortran source code, and the simulator's runtime error messages), plus seven workflow files and one configuration file. A suite of 59 pass-or-fail checks covered five tasks: steady-state flow, transient flow with pumping, coupled flow and solute transport, coupled flow and heat transport, and validation of an existing model. On the unmodified plugin, 52 of 59 checks passed (88%); the seven failures were all runtime errors, six of them caused by constraints spanning more than one input file. Two independent improvement strategies followed: an expert-guided batch correction and an automated one-failure-at-a-time loop. Both reached 59 of 59 (100%) and modified the same four skill files; the expert-guided strategy, which identified five root causes, made one additional edit to a workflow command file. This convergence indicates that the failures came from gaps in the reference files, not from random variation in the model's output. The automated loop required no human intervention. The methodology has been replicated on a sibling simulator (MODFLOW USG-Transport; Krčmář et al. 2026b) with 100% first-attempt success. The result is a reusable design pattern for adapting an LLM coding agent to a scientific simulator without retraining the model.
Dávid Krčmář, Kamila Hodasová, Martin Zatlakovič et al.· Ground Water· 0 citations
As collaborative robots have entered domains such as manufacturing, agriculture, and healthcare, programming or adapting robot behavior typically requires robotic expertise that most end users lack. Natural language lowers this barrier. Recent advancements in large language models (LLMs) have made it feasible to translate natural language into robot task plans. However, language-based task specification suffers from semantic ambiguity, and generative models lack transparency for how language instructions become robot actions, making it difficult for users to validate the plan before execution. To address these issues, we introduce SHRIMP, a system that allows users to automatically generate a hierarchical robot primitive plan using natural language and iteratively revise their plan through re-prompting and explicit correction. At each revision, SHRIMP allows users to validate their plan in simulation, and once satisfied, execute it on the physical robot. Through a user study involving participants planning tabletop kitchen tasks (n=35), we validate that SHRIMP improves perceived control and enhances robot transparency. System videos and source code are available at https://wisc-hci.github.io/SHRIMP.
Mya Schroder, Yuna Hwang, Callie Y. Kim et al.· 0 citations
MaCoPlanner is presented, a task-planning framework built on knowledge compiled from equipment manuals that converts equipment manuals into a typed intermediate representation, retrieves task- and state-relevant evidence, and uses it to support plan generation.
Gui-Peng Xin, Jiahe Xua, Mohammad Deghat et al.· 0 citations
This paper presents LVR-Draw, a fully local language–vision pipeline for robotic drawing that integrates structured scene generation, multimodal verification and correction, and deterministic execution within a unified Human–AI–Robot loop. Given a natural language prompt, a Large Language Model (LLM) generates a structured scene representation in a predefined format, which is rendered into an interpretable image. A Vision–Language Model (VLM) then performs visual inspection to detect inconsistencies in object placement and spatial relationships. These observations are processed by the LLM to produce structured editing operations, enabling iterative refinement of the scene. After validation, the refined scene is converted into executable robot instructions through a deterministic pipeline, supporting predictable and reproducible execution without using generative models for control. Experimental results suggest that the system can generate valid scene representations, support multimodal correction, and preserve drawing order during physical execution.
Nuttasorn Aiemsetthee, Renke Liu, Kave Salamatian et al.· 2026 IEEE International Conf...· 0 citations
Advances in large language models (LLMs) have sparked interest in automating parametric CAD modeling through natural language. Existing LLM-based approaches often treat CAD modeling as flat text generation, overlooking the hierarchical structure and geometric constraints inherent in CAD programs. We present CAD-Factory, a Text-to-CAD generation system for language-driven CAD modeling that explicitly models the structural and parametric semantics of CAD programs. Our core contribution is a new formulation of CAD generation as structured program synthesis, coupled with a learnable hierarchical CAD program representation that disentangles structural topology from parameter instantiation. Building on this formulation, CAD-Factory adopts a manager–programmer–reviewer architecture: a planning agent infers program hierarchy, a coding agent instantiates symbolic and numerical parameters, and an evaluation module enforces structural validity and geometric feasibility, which supports structure-aware reasoning, constraint-consistent generation, and interpretable program synthesis. An editor–viewer loop further enables part-aware code refinement through visual feedback, supporting iterative and controllable design workflows. We also contribute a process-oriented annotation pipeline and a Text–CAD dataset with parameter-free or partially specified descriptions that reflect real-world design expression. Extensive experiments demonstrate that CAD-Factory significantly improves structural correctness and geometric consistency across diverse CAD code generation and editing tasks, establishing a structured, interpretable, and robust framework for AI-assisted design.
Yang Liu, Daxuan Ren, Yijie Ding et al.· International Conference on...· 2 citations· ⚡2
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
