From expert consensus to large language models: What information is needed to determine and justify a contextualized smallest effect size of interest?

Abstract

As psychological science is shifting from traditional null-hypothesis significance testing toward interval-based hypothesis testing, researchers have to determine and justify what the smallest effect size is that matters practically or theoretically, also known as the smallest effect size of interest (SESOI). However, researchers frequently struggle to determine contextualized SESOIs, possibly due to the lack of a general framework outlining the necessary information to do so. In this article, we propose a general framework including the information needed to determine and justify contextualized SESOIs across psychological research. In Study 1, we empirically examined this framework by surveying 57 memory experts using a hypothetical study and asking them to determine a SESOI. The majority indicated that the general framework provided sufficient information to make and justify this decision. However, conducting expert consensus studies for every new study is resource-intensive and impractical. Thus, in Study 2, we developed a large language model (LLM) approach and an application (https://paulriesthuis.github.io/SESOI-generator/) to assist this process. The LLM approach generated a similar SESOI estimate and cost-benefit justifications to those of the memory experts in Study 1. Ultimately, while researchers must critically evaluate and adapt LLM outputs to avoid automation bias, this approach provides a highly practical, scalable starting point to help researchers determine and justify SESOIs, which can facilitate the usefulness of interval-based hypothesis testing.