Can Large Language Models Decompose User Stories into Tasks? Exploring the Role of Prompting Strategies and Models

Abstract

Refining user stories (USs) into tasks is an important step in agile software development, as it translates software requirements into project management artifacts that are assigned to team members. This story-to-task decomposition can be tedious. In this paper, we propose and evaluate DELTA, a Large Language Model (LLM)-based approach that automatically generates tasks from USs and assesses the quality of the resulting decompositions. We conduct a two-phase empirical study to evaluate DELTA. In the first phase, we identify the best-performing DELTA configuration by comparing different prompting strategies, LLM families, and sizes using text similarity metrics on human-created and LLM-generated decompositions. The comparison based on these metrics did not identify a single consistently best-performing configuration. Therefore, in the second phase, we conducted an expert evaluation to determine whether the observed differences in text similarity are reflected in expert judgments and whether DELTA’s criteria-based evaluation module, which uses LLM judges, aligns with expert judgments. Our results show that DELTA’s evaluation module produces rankings that align more closely with expert judgments than text similarity metrics. Moreover, experts generally prefer the top three LLM-generated decompositions, as ranked by LLM judges, over those created by humans, and they express a willingness to adopt DELTA.