Towards Reliable Research: Development and Medical Verification of a Health Claims Corpus

Abstract

The proliferation of digital information has fundamentally changed health-seeking behavior, with individuals increasingly turning to the internet for medical advice. This trend, evidenced by millions of search queries on Runet, is complicated by highly variable reliability of online health information. While Natural Language Processing (NLP) offers promising solutions for automated credibility assessment, its development is hindered by the scarcity of high-quality, expertly annotated datasets. Existing resources often lack in medical verification, being labeled through crowdsourcing or by non-specialists, which severely limits their practical utility. This paper presents a novel Russian-language dataset of health claims and details its meticulous development methodology. The dataset construction followed three core principles: representativeness (reflecting actual Runet discourse), balance (across truthfulness and valence), and verifiability (against authoritative sources). A significant innovation is the inclusion of a “Disputed” category for claims involving partial truths, exaggerations, or insufficient evidence. A multi-stage verification pipeline involved linguistic screening by philologists, independent assessment by six physicians (Fleiss’ κ = 0.316), and source-backed annotation by medical experts (κ = 0.383). A journalist refined formulations for accessibility, while a meta-reviewer resolved discrepancies based on argument persuasiveness. The final dataset contains 108 claims (40 True, 40 False, 28 Disputed) across five themes: Vaccination, GMOs, Lifestyle, Alternative Medicine, and Vitamins. Beyond medical NLP applications, this resource will be used/employed in few-shot learning, psycholinguistics, and AI verification strategies. The study demonstrates that reliable medical data annotation requires expert-driven, multi-layered validation with mandatory source verification, emphasizing that linguistic precision is as crucial as factual accuracy. Thus, the developed dataset makes a valuable contribution to both computational linguistics and digital healthcare research, providing a robust benchmark for future studies in Russian-language medical text processing.