Use of Thresholds for Rating Certainty of Evidence with the GRADE Framework: A Systematic Survey of Systematic Reviews.
Abstract
Objective
Current guidance regarding applying the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) framework mandates establishing thresholds to assess certainty of evidence. These thresholds include the null, the minimally important difference (MID), and moderate and large effect thresholds. GRADE experts offer a rationale for use of each particular threshold. However, GRADE methodologists continue to debate the optimal choice of threshold, particularly whether to use the null threshold. We aimed to describe how systematic review authors used thresholds and in doing so to provide insight into the perceived utility of the available thresholds and to inform ongoing debate regarding the optimal choice of threshold. STUDY
Design
AND
Setting
We conducted a systematic survey sampling the 200 most recently published Cochrane and 200 non-Cochrane systematic reviews that used GRADE, which we identified from the Cochrane Database of Systematic Reviews (2018 to July 2024) and MEDLINE (2018 to December 2024). We documented whether review authors used thresholds and the thresholds they chose.
Results
Among the sampled reviews, 118 of 200 (59%) Cochrane and 63 of 200 (31.5%) non-Cochrane reviews used thresholds. Among the reviews with an identifiable threshold type, 55 of 112 (49.1%) Cochrane and 26 of 55 (47.3%) non-Cochrane reviews used the null, while 54 of 112 (48.2%) Cochrane and 25 of 55 (45.5%) non-Cochrane reviews used the MID. Few reviews used the MID, moderate, and large effect thresholds together to form ranges of effects (2 of 112 [1.8%] Cochrane; 3 of 55 [5.5%] non-Cochrane).
Conclusion
Use of thresholds in systematic reviews using the GRADE framework remains suboptimal, particularly in non-Cochrane reviews. For systematic reviews that do use thresholds, reviewers choose the null and the MID equally and very seldom use multiple thresholds, suggesting the perceived utility of the available options.