Compiling Large Multi-modal Requirement Documents into Runnable Software Systems: From an Agentic Test-Driven Perspective

Abstract

Large Language Models (LLMs) have significantly improved programming efficiency by parsing natural language into code snippets. However, their performance degrades significantly as requirements scale; when faced with multi-modal documents containing hundreds of scenarios, LLMs often produce incorrect implementations or omit crucial constraints. Observing LLMs' ever-evolving capability and their persistent stochastic hallucination, we raise a question: whether it is possible to make LLM-based agentic programming go beyond "code generation" to "requirement compilation", i.e., whether programmers can produce a runnable system by only accomplishing (non-trivial) requirement documents? In this work, we take a first step by proposing the ARC (Agentic Requirement Compilation) technique to parse a multi-modal requirement document, describing hundreds of scenarios in a DSL format, into a runnable software system. In addition to the source code, ARC also generates software engineering artifacts including (1) a modular design that spans the user interface, API interface, and database, (2) enriched test cases for each interface (including unit tests, modular tests, and integration tests), and (3) detailed traceability across all artifacts for software maintenance. Our approach employs a bidirectional test-driven agentic loop: (1) a top-down architecture phase that decomposes requirements into UI, API, and database interfaces, each of which is equipped with verifiable test suites, and (2) a bottom-up implementation phase where agents generate code that must satisfy the generated tests. Throughout this process, ARC maintains strict traceability across requirements, design, and code to facilitate intelligent asset reuse and follow-up maintenance. We evaluate ARC on two complementary benchmarks, i.e., a depth-oriented benchmark of 6 runnable web systems spanning 50-200 requirement scenarios, and the breadth-oriented AppForge benchmark comprising 101 Android app generation tasks. Across 3 independent trials, ARC outperforms all state-of-the-art LLM-based baselines, with the generated web systems passing on average 50.6% more GUI tests, and achieving 100% compile success and 68.3% test case pass rate on AppForge. In addition, a user study with 21 participants shows that participants with limited programming experience successfully write DSL-based documents consisting of 50 to 174 scenarios, within 5.6 hours on average, to generate a runnable system such as a real-world ticket-booking system of around 10K lines of code with maintainable architecture.