Webis-SR4ALL-26: A cross-disciplinary corpus of systematic reviews

Abstract

Overview Webis-SR4ALL-26 is a large-scale corpus of 301,871 systematic reviews spanning 27 scientific disciplines. The corpus is derived from OpenAlex and enriched with structured methodological information extracted from full texts where available. The dataset is designed to support research on systematic review retrieval, screening, and meta-research across domains. It provides a unified, open infrastructure for large-scale experimentation under consistent indexing and citation conditions. This release accompanies the paper: TBD Files Included 1. sr4all_full.jsonl This file contains 301,871 systematic reviews in JSON Lines format (one JSON object per line). Each record includes: OpenAlex Metadata (available for all reviews) id (OpenAlex ID) title doi abstract (if available) year type source cited_by_count referenced_works_count referenced_works (OpenAlex IDs) pdf_url (if available in OpenAlex) language field subfield topics keywords authors All citation links are derived exclusively from OpenAlex indexed reference lists. No external reference parsing or enrichment was performed. Structured Methodological Fields (subset only) For reviews where full text was available and successfully parsed, additional fields may be present: objective research_questions keywords_used exact_boolean_queries inclusion_criteria exclusion_criteria n_studies_initial n_studies_final year_range year_range_normalized databases_used snowballing These fields are only present where full text was available and the extraction pipeline produced verified outputs. Missing fields are represented as null or absent. Full texts themselves are not included in this release due to copyright restrictions. Where available, pdf_url links provided by OpenAlex are included for research access. 2. sr4all_normalised_queries.jsonl This file contains normalized Boolean queries derived from reported search strategies. Each record includes: id (OpenAlex ID; maps directly to sr4all_full.jsonl) boolean_queries (list of normalized Boolean expressions) Normalization removes database-specific syntax and retains: Topical terms Logical operators: AND, OR, NOT Temporal constraints and metadata filters are not embedded in the Boolean string and must be applied externally. Only reviews with extractable and normalizable search strategies are included in this file. Queries that could not be normalized are omitted. Corpus Construction Summary Initial candidate retrieval from OpenAlex using title-based heuristics. Deduplication via DOI, OpenAlex ID, and normalized titles. Metadata filtering (English language, DOI present, non-update reviews). Integration of benchmark systematic review datasets. Final corpus size: 301,871 systematic reviews. Full-text PDFs successfully retrieved and parsed: 72,678. Structured methodological extraction applied to parsed full texts. Extraction follows a verify-then-repair pipeline prioritizing precision over recall. Format Both files use JSON Lines (.jsonl) format: UTF-8 encoding One JSON object per line No trailing commas Suitable for streaming and large-scale processing. What This Corpus Enables The corpus supports multiple lines of research: 1. Retrieval Evaluation Execute normalized Boolean queries against OpenAlex. Compare retrieved results against cited reference sets. Compute precision, recall, and F-scores. Analyze cross-domain retrieval behavior. 2. Screening and Ranking Use reference sets as weak supervision for relevance modeling. Benchmark citation screening methods at scale. Study cross-domain generalization of screening models. 3. Query Generation and Reformulation Learn query structure from extracted expert-written strategies. Compare LLM-generated queries against author-reported strategies. Analyze logical structure across disciplines. 4. Meta-Research and Science-of-Science Study reporting practices (objectives, inclusion criteria, databases). Analyze disciplinary differences in search strategy formulation. Examine temporal trends in systematic review methodology. 5. Large-Scale Observational Analysis Characterize differences in methodological rigor across fields. Study variation in Boolean complexity and vocabulary. Analyze inclusion criteria formulations across domains. Important Limitations Full texts are not distributed due to copyright restrictions. Structured fields are only available for reviews with accessible and successfully parsed PDFs. Extraction prioritizes precision; some valid information may remain unstructured. Reference sets reflect cited works, not strictly included-study gold standards. Normalized queries are simplified approximations of original database-specific queries. Intended Audience Information retrieval researchers Systematic review automation researchers Bibliometrics and science-of-science researchers LLM researchers working on grounded extraction and query generation Developers of evidence synthesis tools