The Validity of AI-Generated Multiple-Choice Questions for Students With Disabilities

Abstract

Classroom assessments must meet the rigorous validity standards required for students with disabilities. This study evaluates whether artificial intelligence (AI) can be a reliable partner in the high-stakes environment of special education data collection by examining the validity of multiple-choice questions generated by three large-language-model chatbots (ChatGPT, Copilot, Gemini). Each chatbot produced 200 items under default and deeper settings (N = 600) and items were scored against 15 objective item-writing guidelines. Models frequently violated rules about complete stems and randomized key placement, with ChatGPT having the highest overall error rate and Copilot and Gemini the lowest. Chi-square and ANOVA analyses showed significant differences across chatbots, especially for specific guidelines, while deeper prompts improved quality only slightly. Results suggest AI can rapidly draft items with solid, plausible distractors, but educators must review and revise them to ensure their quality.