Detecting Software Requirements Smells Through Domain-Augmented Generation

Abstract

Context: The quality of software requirements is a critical determinant of success in mission-critical and large-scale software systems. Requirements smells, which indicate potential quality issues in software requirements, such as imprecision, ambiguity, and poor verifiability, can significantly impact project outcomes. Despite their crucial role in software projects, software requirement documents often contain inconsistencies, ambiguities, and conflicts that are challenging to detect and manage manually. Objective: We aim to develop a novel approach using data representation and a large language model (LLM) based approach to improve the detection of conflict and ambiguity requirements smells. Method: We construct domain-specific vector databases from software requirements and domain knowledge documents, where vectors represent embedded requirements and contextual domain information. These databases are populated offline and utilized during inference time for semantic similarity-based retrieval and smell detection. We compare our domain-augmented generation based approach with traditional reasoning-based methods for performance evaluation using diverse retrievers. Additionally, we include zero-shot and one-shot prompt techniques. Results: Our experimental results demonstrate that our RAGbased approach detects requirements smells across multiple datasets using five LLMs: Claude 3.5 Sonnet, Claude 3.7 Sonnet, GPT-4o, o3-mini, and GPT-5. For conflict smell detection, compared to previous approaches on cross-datasets, these models have superior F1 scores, up from 0.566-0.804 to 0.676–0.857. The integration of domain knowledge showed mixed results, generally improving recall while decreasing precision and F1 scores. For ambiguity smell detection, the GPT-5 model achieved an F1 score of 0.783 compared to 0.762 of previous approaches. LLM-based approaches, besides GPT-4o, had better precision scores compared to previous approaches. Domain knowledge integration substantially increased the F1 score of the tested LLMs, with the most significant improvement observed in Claude 3.7 Sonnet, which rose from 0.526 to 0.677. In contrast, misleading or low-quality domain knowledge decreased F1 scores. Conclusions: The proposed RAG-based approach offers a promising automated solution for detecting requirements smells, demonstrating competitive performance compared to existing methods. Integrating domain knowledge enhances detection capabilities while highlighting the importance of balancing precision and recall.