Appraisal of Recent Intervention-Focused Umbrella Reviews Reveals Major Methodological Weaknesses: A Meta-Epidemiological Study.

Background

Umbrella reviews (URs) are frequently used to provide high-level syntheses of evidence to inform clinical practice, policy, and guideline development. However, the methodological quality (MQ) and adherence to reporting standards of recent intervention-focused URs is understudied.

Objective

To evaluate a random sample of recently published URs of interventions in health/medicine across several potential domains of MQ. Secondary objectives were to evaluate adherence to the Preferred Reporting Items for Overviews of Reviews (PRIOR) guideline and other recognised methodological guidance (e.g., Cochrane Handbook, JBI Manual).

Methods

PubMed, Cochrane Library and Epistemonikos were searched in January 2026 for URs of interventions. We selected a random sample of 100 URs in health and/or medicine published between January 2024 and January 2026. Data were extracted pertaining to the following domains: identification and selection of systematic reviews (SRs), overlap of primary studies, bias inherited from included SRs, data extraction and evidence mapping, synthesis and interpretation bias, and reporting and transparency.

Results

Of the 100 URs, 69% lacked a complete search strategy (median=4, IQR: 3-6 databases). Eligibility criteria were insufficiently reported in 96% of URs. Quantification of overlap was unclear in 77%, and only 11% considered overlap in synthesis. Only 21% of URs adhered to PRIOR. AMSTAR II was used in 77%, with AMSTAR I and JBI at 5% each. In 48%, inherited risk of bias from the SRs was high. Only 9% described how outdated or superseded SRs were managed and none managed discordant review conclusions. Outcome definitions were inconsistent in 71%, while only 9% avoided selective extraction of outcomes or analyses. Handling of multiple meta-analyses was unclear in 96%, and 71% selectively emphasised statistically significant results. Only 25% clearly described handling of heterogeneity and small-study effects, and 24% used evidence grading. 19% failed to register a protocol, and only 10% justified deviations from planned methods. Finally, 84% did not explicitly acknowledge umbrella-level limitations.

Conclusions

We identified multiple methodological weaknesses in recent URs of interventions. Existing assessment tools appear only partially fit for purpose to assess the quality of the URs. Adherence to available guidance during the conduct and reporting of these recent URs is poor.