Exploring the Feasibility of Machine Learning-Based Automated Scoring for Constructed-Response Assessment in Technology Education

Abstract

As educational paradigms shift from knowledge-based to competency-based approaches, the importance of constructed-response assessment has increased because it enables comprehensive evaluation of students’ thinking processes and problem-solving abilities. However, scoring constructed-response items requires substantial time and effort, resulting in a heavy workload for teachers and difficulties in ensuring scoring consistency. Although machine learning-based automated scoring has recently attracted considerable attention as a potential solution, research focusing on technology education remains limited. Therefore, this study developed an assessment instrument and scoring rubric for constructed-response assessment in technology education and examined the feasibility of applying a machine learning-based automated scoring model by designing both morpheme-based and keyword-based scoring features from student response data. A total of 548 responses collected from students at six middle schools were used for analysis. Hybrid scoring features combining morpheme-based and keyword-based features were applied, and the performance of various machine learning algorithms was compared and analyzed. The results showed that the hybrid features effectively captured both the structural and semantic characteristics of students’ written responses in technology education, and that automated scoring performance varied according to the conceptual and problem-solving demands of each assessment item. Furthermore, SHAP-based explainable AI analysis revealed that the model relied not only on key conceptual terms but also on linguistic features such as response length and sentence structure when making scoring decisions. However, the model tended to be more sensitive to the richness and structural completeness of responses than to conceptual accuracy or logical validity. These findings demonstrate the feasibility of applying machine learning-based automated scoring to constructed-response assessment in technology education through appropriate scoring feature design and high-quality training data. They also suggest the need for future research on hybrid automated scoring models that integrate semantic features with large language models (LLMs).