Beyond Blind Trust in Hospital Data Warehouses: Evaluating Real‐World Data Accountability

Abstract

ABSTRACT Introduction The reliability of real‐world evidence (RWE) depends on data accuracy and secondary users' ability to evaluate how data are generated and transformed. Although recent frameworks emphasize provenance, transparency, and auditability, their practical assessment in routine hospital settings remains unclear. This study examines the practical evaluability of real‐world data (RWD) and explores operational boundaries limiting data quality assessment within learning health systems. Methods An empirical case study was conducted at a tertiary‐care university hospital in Japan. Data extracted from electronic health records and a clinical data warehouse were compared using 15 quality indicators derived from the Data Quality Management Guidebook of the Digital Agency of Japan (based on ISO/IEC 25012). Evaluability was assessed using a two‐step framework comprising empirical verification and contextual explanation from the perspective of research support personnel operating under routine institutional access permissions. Results Evaluability varied substantially across clinical domains. Diagnosis records were generally verifiable and explainable, whereas medication‐related domains contained observations that could be verified but not fully explained by secondary users. Indicators including currentness and efficiency faced additional operational constraints, revealing boundaries between observable data characteristics and available explanatory information. Conclusions Practical RWD use depends not only on observable data characteristics but also on the accessibility of information needed to interpret them. Evaluability boundaries reflect operational constraints rather than poor data quality. Rendering these boundaries visible may support more transparent and accountable RWD governance and inform future improvements through system design and data governance practices.