Evaluating an Iteratively Revised Assessment for a Global Pediatric Surgery Short Course Using Mixed-Effects Modeling and Item-Level Rasch Analysis.
TL;DR
A single-day pediatric surgery short course was associated with significant improvements in postcourse knowledge across all domains, while iterative assessment revisions preserved score comparability across versions and sites.
Abstract
Objective
To evaluate changes in learner performance after a global pediatric surgery short course, measure consistency of scores across 4 iteratively revised assessment versions, and examine item-level performance and language-related bias.
Design
Prospective pre- and postcourse assessment study with generalized linear mixed-effects models for whole-test and domain-level analyses and many-facet Rasch modeling for item-level analyses and language bias.
Setting
Five rural hospitals (2 districts, 3 regional) in Rwanda (January-March 2025).
Participants
Sixty nonspecialist physicians (26 interns, 23 junior physicians, 11 senior physicians). Most preferred English (n = 54), with a minority preferring French (n = 6). Two-thirds had prior pediatric surgery exposure (n = 40).
Results
Overall scores improved significantly from pre- to postassessment, from 71.3% to 86.5% (+15.2 percentage points; p < 0.001). Across domains, mean percentage correct increased from 73.4% to 94.0% in Diagnosis, 69.7% to 83.4% in Transfer, and 70.9% to 84.7% in Management (all p < 0.001; partial eta squared 0.47-0.88). Across the 4 assessment versions, there were no significant differences in baseline scores, postcourse scores, or pre- to postcourse change for total or domain-specific scores (p > 0.05). In mixed-effects models, language preference was associated with total scores and the Management domain (p = 0.03-0.05; partial eta squared, η²p ≥ 0.41), while training level, prior pediatric surgery exposure, hospital site, and assessment version were nonsignificant (all p > 0.05). Item-level analyses identified items with misfit and language-related differential item functioning, including bias disadvantaging French-preferring learners for several items.
Conclusion
A single-day pediatric surgery short course was associated with significant improvements in postcourse knowledge across all domains, while iterative assessment revisions preserved score comparability across versions and sites.