Skip to content
Book Open access

Break It First: Stakeholder Participation in Quality Assurance for LLM Features

Oct 2026 · Adjunct Proceedings of the 14th Nordic Conference on Human-Computer Interaction · 0 citations · 7 references

Abstract

Large language model (LLM) features are difficult to predict upfront. Their useful behavior, failure modes, and acceptable boundaries often become visible only through situated interaction. This Industry Experience paper reports a stakeholder-led quality assurance process used in the development of an LLM-enabled website search assistant. Stakeholders first described expected and unwanted behaviors, then deliberately tested the artefact to find errors before wider deployment. The process produced 44 behavioral specifications, 29 unwanted behaviors, 13 testing targets, and 26 issue reports, including content errors and contextual risks that depended on stakeholder knowledge of the operating environment. We argue that LLM quality assurance should treat stakeholder participation not only as acceptance testers but as a practical adjustment space for discovering situated errors and refining system behavior.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.