Decision Support in Publicly Available Patient Information Policies at U.S. Osteopathic Medical Schools: A Vignette-Based Document Analysis

Abstract

Research Objectives: To evaluate whether publicly available institutional guidance supports patient-information decisions across scenarios and schools, and characterize document synthesis and evaluation consistency. Methods: We conducted an exploratory, vignette-based document analysis of a geographically diverse nonprobability sample of 20 U.S. osteopathic medical schools. Eight educational vignettes yielded 160 school-vignette pairs. Each pair underwent three separate AI-assisted retrieval-and-evaluation runs, classifying decision support as Explicitly Supported, Inferable, Ambiguous, or Not Addressed. Response selection prioritized greater support for discordant pairs, followed by fewer contributing documents and run order. One investigator verified or revised selected discordant classifications against cited evidence. Explicitly Supported and Inferable were grouped post hoc as sufficient decision support. Analyses were descriptive and included an exploratory two-school model-investigator comparison. Results: At least one eligible source was retrieved for 159 of 160 pairs (99.4%). Final classifications were Explicitly Supported for 18 pairs (11.3%), Inferable for 3 (1.9%), Ambiguous for 132 (82.5%), and Not Addressed for 7 (4.4%). Sufficient support occurred in 21 pairs (13.1%), most frequently for generative AI-assisted reflective writing (7/20 schools, 35%), and in none for personal cloud notes or official clinical logs. Ten schools had no sufficiently supported vignette; the maximum was four of eight. Multiple documents contributed to 111 evaluations (69.4%). Three-run ratings were unanimous for 113 pairs (70.6%), with 80.2% pairwise exact agreement. Investigator review retained 44 of 47 selected discordant ratings and revised three upward. In the two-school comparison, the investigator more frequently judged evidence sufficient when models judged it insufficient than the reverse. Conclusions: Relevant public guidance was frequently retrieved, but few school-vignette pairs were classified as providing sufficient scenario-specific decision support. These findings highlight a gap between identifying relevant guidance and determining an appropriate course of action within this evaluation framework. They support institutional review of how student-facing materials explain information use, storage, sharing, and approval requirements. Vignette-based review identifies questions requiring clarification. Evaluation with learners and assessment of internal and clinical-site guidance would help determine how these findings translate to students' decisions.