Deploying large language models for AI-driven workflow generation in a production enterprise platform is benchmarked across 29 real-world IT automation scenarios, two generation pipeline architectures, and eight independent runs per prompt-model-pipeline configuration.
Abstract
Enterprise compliance management requires rapid adaptation to evolving regulatory frameworks (e.g., DORA, AI RMF, FedRAMP) and tight remediation SLAs. Traditional static orchestrators often fail in hybrid cloud environments where event-driven assessments demand that automation code adapt to runtime context in seconds. This paper presents lessons learned from evaluating six large language models for AI-driven workflow generation in a production enterprise platform, benchmarked across 29 real-world IT automation scenarios, two generation pipeline architectures, and eight independent runs per prompt-model-pipeline configuration (2,784 runs total). Our initial pipeline used monolithic workflow generation, achieving 31.5-82.8% structural success rates (JSON schema validity and correct UI rendering), with most models struggling on complex JSON generation. We developed a redesigned piecewise pipeline that decomposes workflow construction into variable scaffolding, base block assembly, and nested block generation, raising structural success to 74.1-97.8% across all models. We analyze production tradeoffs including cost (USD 0.008-0.20 per workflow), latency (under 50s for interactive use), and model selection. Piecewise decomposition enables smaller models (e.g., mistral-small at 95.7% structural success and USD 0.01 per workflow) to reach production viability, removing dependency on expensive frontier models. While mistral-medium-2505 and gpt-oss-120b achieved the highest structural success (96.1% and 97.8%), mistral-medium-2505 carries a 19x cost premium versus mistral-small. Our deployment lessons highlight the need to separate structural validity from semantic correctness (logical fulfillment of user intent) and provide a solution for model-agnostic, scalable automation in cloud engineering.
This work proposes CURATE - Composition, User-in-the-loop, Reuse, and Automated Task Execution - a novel human-in-the-loop multi-agent system that uses LLM agents to manage and develop composable workflows across their entire lifecycle.
Nolan Cutler, Chia-Chen Kuo, Nanda Velugoti et al.· 0 citations
This study investigates whether a frontier LLM can generate Dockerfiles and Docker Compose configurations for multi-service applications using repository contents without access to developer-authored deployment artifacts and analytically derives a minimal explicit deployment specification for information that cannot be reliably inferred from repository artifacts.
Oleg Grynets, Kyrylo Fursov, V. Lyashkevych et al.· arXiv.org· 0 citations
Mass customization demands manufacturing systems that can rapidly reconfigure production workflows for diverse product variants. Traditional robot task programming requires specialized engineers and hours of manual coding, which slows agile changeover. We present a low-code task orchestration platform that uses large language models (LLMs) to generate structured manufacturing task plans from natural language descriptions. The system introduces three design choices: (1) domain-aware orchestration generation with structured JSON schema constraints that decomposes natural language into executable subtask sequences with behavior tree skeletons; (2) a generate-validate-feedback closed loop with dual-layer validation (requirement coverage analysis and behavior tree structure/semantic/executability verification) that enables iterative quality improvement; and (3) a standardized 7-step Model Context Protocol (MCP) tool chain that bridges LLMs with the manufacturing execution system. Experiments on 87 template-derived manufacturing tasks across five industrial scenarios demonstrate 98.9% subtask coverage and 98.9% safety step recall with 17.0±4.2s average generation time. On the public BTGenBot dataset (50 samples), our zero-shot approach with Claude Haiku achieves 72% behavior tree validity, compared to ~67% for the fine-tuned BTGenBot baseline (LlamaChat-7B with syntactic auto-correction); note that Claude Haiku is a substantially larger model, so this comparison reflects the combined effect of model scale and prompt design. The validation feedback loop recovers requirement coverage from 18% to 100% in a single iteration. The MCP tool chain achieves 100% end-to-end success across 15 trials. The system is operational on an industrial internet platform, supporting agile changeover in mass customization manufacturing.
Yanping Deng, Jinze Yu· 2026 IEEE International Conf...· 0 citations
Orchestration frameworks for Large Language Models (LLMs) such as LangChain have accelerated the development of AI-driven applications by offering quick administration, retrieval pipelines, tool integration as well as agentic workflows. These abstractions significantly reduce the effort of development while prototyping, but their effectiveness in dealing with enterprise-scale production workloads has not been fully evaluated. This paper systematically investigates the operational limits of LangChain in production environments by undertaking a systematic experimental benchmarking of Retrieval-Augmented Generation (RAG) pipelines, multi-step chains along with agent-driven workflows under concurrent workloads. The study examines LangChain in terms of many other production engineering metrics such as scalability, latency amplification, reliability, observability, fault tolerance and cost efficiency. Experimental results show high cost of orchestration abstraction in high-concurrency scenarios, notably for processes that need sequential reasoning, external API calls, and dynamic tool selection. The findings reveal that limited native observability, dependency propagation, and non-deterministic execution patterns impede debugging, performance optimization, and operational maintenance. The work includes architectural analysis as well as production-oriented implementation, backed by a GitHub repo that provides repeatable benchmarking pipelines, failure injection testing, distributed tracing instrumentation, and deployment-ready RAG processes. The evaluation demonstrates that LangChain is highly effective for rapid AI application development; however, enterprise-scale deployment requires substantial enhancements. The paper states that LangChain should be thought of as an orchestration accelerator and not a fully functional runtime framework. For LLM-based systems to be broadly adopted by corporations, we need to improve deterministic execution, runtime tracing, orchestration efficiency, and production observability.
Madhurima Kommuru· American International Journ...· 0 citations
This work introduces a specialist workflow for the transformation of Business Process Model and Notation diagrams into executable agentic workflows and finds that generalist agents generate code inconsistently in both functionality and quality, limiting their suitability for industrial settings where reliability and maintainability are essential.
Harris Borman, Herman Wandabwa, Fu-Sun Yu et al.· arXiv.org· 0 citations
DataClawEval is introduced, the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios, and it comprises 100 rigorous, end-to-end tasks spanning five execution engines.
