When may LLM outputs influence software requirements? A human-in-the-loop governance framework

Abstract

Large language models are increasingly used to review, clarify, rewrite, and trace software requirements. These applications create a governance problem that output-quality assessment alone cannot resolve: a fluent proposal may rely on inadmissible evidence, alter stakeholder intent, introduce unsupported specificity, or imply an organizational commitment that the model has no authority to make. Existing work on retrieval-augmented generation, controlled natural language, formal verification, human oversight, and AI governance supplies relevant controls, but it does not specify the procedural status of an individual LLM proposal relative to a controlled requirements artifact. This article develops an artifact-centered, human-in-the-loop framework in which the permitted influence of a proposal is the primary object of governance. The framework combines five governance functions—governed evidence, bounded context construction, controlled LLM analysis, pre-commit verification, and accountable human approval—with four artifact-influence states: A0 advisory observation, A1 evidence-linked candidate, A2 verified recommendation, and A3 approved and committed change. Its central theoretical claim is that output quality, evidential legitimacy, verification status, and authority to commit a change are distinct properties and should not be collapsed into a single confidence judgment. Seven falsifiable hypotheses translate the model into measurable comparisons involving source admissibility, context leakage, unsupported specificity, semantic drift, reviewer agreement, unreviewed changes, governance cost, and organizational maturity. Human review is treated as both a necessary decision boundary and a potential source of automation bias, anchoring, and fatigue. The framework is conceptual rather than empirically validated and provides a basis for controlled experiments, field studies, and longitudinal evaluation.