Probing the Crowd Online: A Semi-Automated Analysis of Explanation Needs in Subreddit Threads

Abstract

If software is complex and difficult to use, endusers increasingly rely on external, often inaccurate information sources. To avoid this, system designers must elicit and implement appropriate explanation requirements, enabling their software to explain itself without external help. The accurate elicitation of explanation requirements is challenging, as the need for explanations is highly subjective. Requirements engineers need access to large amounts of user feedback to frame and prioritize explanation needs across stakeholder groups. This can be facilitated using data mining and related CrowdRE techniques. While contemporary research has focused on app reviews to elicit explanation needs, social networks as a data source remain largely unexplored. This work reports an analysis of posts from four softwarefocused Reddit forums. We examined the posts for primary and secondary explanation needs, and for the system aspects that these needs relate to. To demonstrate scalability, we show that LLMs (in our case, GPT-5.1) can label these explanation needs with reliability comparable to that of expert human coders. We provide two major contributions: 1) an analysis of 5152 Reddit posts labeled for explanation needs, and 2) the guidelines and software necessary to replicate our work and adapt it to similar research contexts.