Reimagining Education: Innovations in Teaching, Learning and AI in Higher Education

Abstract

Universities are being offered artificial intelligence that flags students likely to fail, and the case for buying it rests on accuracy figures drawn from papers that split one pooled pile of student records at random. This study asks how much of that accuracy a university would actually see. Using the Open University Learning Analytics Dataset, 32,593 student registrations across seven modules and twenty-two presentations with 10,655,280 recorded interactions, a gradient-boosted model was trained to predict failure or withdrawal from what was known on day 30 of a course. Under a random split it reaches an area under the ROC curve of 0.8450. Three sources of inflation were then separated on identical test rows. Losing familiarity with the course costs almost nothing, 0.0085 of area under the curve, consistent in direction across 17 of 22 presentations but small by every effect size used (Wilcoxon p = 0.00219, Cliff's delta 0.223, bootstrap interval spanning zero); training on 2013 and predicting 2014 costs 0.0037 and is not significant (p = 0.12720). What does matter is arithmetic rather than transfer. Pooling students across presentations that fail at rates from 0.274 to 0.658 adds 0.0267 that no single course ever sees, and a model told only which presentation a student is enrolled on, nothing about the student, already reaches 0.5777. Larger still, 15.7 percent of students had already unregistered before day 30; removing them costs 0.0575. The model is better at recognising students who have already left than students who will fail while trying (0.8799 against 0.7948). At an alert budget of one student in five it misses 58.6 percent of those at risk, and misses the least deprived more often than the most deprived, a pooled gap of 0.118 that shrinks to 0.086 and reverses in one module once deprivation is compared within courses. Applied together, an unseen course and a population restricted to students still enrolled bring the figure to 0.7520, a fall of 0.0931 of which 62.2 percent is departed students, 28.7 percent is pooling and 9.1 percent is the unfamiliar course. The paper concludes that reported accuracy in this literature is largely a property of how the test set was assembled, and recommends within-course reporting, the exclusion of students already gone, and a stated alert budget