Evaluating Gemma 3 27B for local undergraduate physics tutoring

Abstract

Abstract Physics education research has developed a range of student-centered approaches that emphasize conceptual understanding and problem solving, creating interest in whether large language models (LLMs) can provide complementary, on-demand tutoring support. Locally deployable models are particularly relevant for institutions seeking greater control over data handling and system operation, but their capability relative to commercial models and their reliability in authentic learning interactions remain insufficiently characterized. We introduce mlphys101, a multilingual multiple-choice benchmark of introductory physics questions, and evaluate Gemma 3 27B on its German variant in three quantization configurations. The locally deployed model achieves high benchmark accuracy, although performance is lower and more variable on conceptual and multi-step reasoning tasks than that of a commercial cloud baseline. We subsequently deploy the same model as a tutoring chatbot in a classroom field experiment with first-semester engineering students. Survey responses indicate that explanations were generally comprehensible, but the practical usefulness of the system was limited by frequent prompt reformulation, occasional incorrect or incomplete responses, difficulties in using information from the ongoing interaction, and response latency. These findings demonstrate that strong performance on a controlled physics benchmark does not necessarily translate into reliable tutoring behavior. Locally deployed LLMs may therefore provide useful support for selected learning activities, but their suitability for educational use should be evaluated not only through domain benchmarks but also under realistic, multi-turn interaction conditions.