Information quality assessment of DeepSeek artificial intelligence on orthodontic aligners: A reliability and validity study

Abstract

The objective of the study is to evaluate the information quality, reliability, and clinical utility of DeepSeek AI’s responses to frequently asked questions (FAQs) about orthodontic aligners. Seventeen orthodontic aligner FAQs were systematically compiled from clinical expertise and public search analysis. Each question was presented twice to DeepSeek AI (version 3.1) with a 1-week interval. A blinded orthodontist evaluated all 34 responses using a modified 5-point Global Quality Scale (GQS). Intra-rater reliability was assessed through weighted Cohen’s kappa and intraclass correlation coefficient (ICC). AI performance was analyzed for proficiency (low/high-threshold criteria), test-retest reliability (ICC), and internal consistency (Cronbach’s alpha). Expert intra-rater reliability was excellent (κ w = 0.89, ICC = 0.92). DeepSeek AI demonstrated exceptional performance: 100% of questions met low-threshold proficiency (GQS ≥4 both rounds), while 94.1% achieved high-threshold proficiency (GQS = 5 both rounds). Test-retest reliability was excellent (ICC = 0.98), and internal consistency was high (α = 0.96). Qualitative assessment revealed perfect scores in accuracy, clarity, and actionability, with minor reductions in completeness (4.97/5) and objectivity (4.94/5) for comparative questions. DeepSeek AI generates high-quality, reliable information for patient education about orthodontic aligners, demonstrating near-perfect proficiency and consistency. While it excels in factual content, minor limitations exist in balanced comparative analysis. These findings support the cautious integration of AI as a supplementary educational tool in orthodontic practice, with professional oversight recommended for clinical discussions.