Preliminary accuracy, inter- and intra-rater reliability of a 4-point cervical artery dissection diagnostic decision support tool in an ambulatory setting using real and simulated cases: a quality improvement project

Abstract

Abstract Background Cervical artery dissection (CeAD) is a rare but life-threatening condition that commonly presents with non-specific symptoms such as neck and head pain, complicating early recognition in ambulatory care settings. The CeAD diagnostic decision support tool (DDST) is a 4-item assessment designed to identify patients that should be referred for advanced diagnostic work-up to rule out CeAD; however, its utility in ambulatory settings has not been established. Methods This quality improvement project aimed to evaluate the accuracy, inter-rater reliability, and intra-rater reliability of chiropractors applying the CeAD DDST using clinical vignettes simulating a typical ambulatory encounter before deploying the DDST broadly as part of the clinical workflow. Five experienced chiropractors assessed 40 standardized clinical vignettes. CeAD-positive cases ( n = 10) were derived from published imaging-confirmed ambulatory care case reports. CeAD-negative cases ( n = 30) were developed using generative artificial intelligence and refined by senior authors to reflect common conditions encountered in chiropractic practice. Raters independently assessed cases with the DDST (score 0–4), with a cutoff score of ≥ 3 recommending referral for advanced diagnostic work-up in two rounds, beginning November 11, 2025 and January 07, 2026, with a two-week capture period. Accuracy, inter-rater reliability, and intra-rater reliability were assessed. Results Accuracy in identification ranged from 72.5% to 95.0% across raters and rounds. False positives and negatives were present across raters in both rounds, with false positives being more prevalent. Inter-rater reliability was moderate to substantial in both rounds (Round 1: κ = 0.609, 95% CI 0.432–0.806, Round 2: κ = 0.540, 95% CI 0.365–0.735). Intra-rater reliability ranged from moderate to almost perfect agreement (range 0.457–0.931). Conclusions The CeAD DDST demonstrated moderate to good accuracy and reliability among chiropractors in a simulated ambulatory setting. The presence of false negatives and inter-rater reliability falling below recommended benchmarks preclude recommendation for implementation at this time. Iterative refinement informed by end-user input and prospective ambulatory testing is warranted prior to clinical adoption of the CeAD DDST. Future evaluations should include prospective studies using real clinical data to better represent real-world settings.