Is there a way to improve language models accuracy in solving physics problems in Russian and English languages?

Abstract

Large language models are increasingly used to solve educational physics problems, but their accuracy remains inconsistent. This study investigates whether prompt formulation can improve the performance of language models on school-level physics problems in Russian and English. In total 21 language models were tested on 25 problems with 3 different methods: original textbook problem statements, simplified statements written in a structured Given/Find format, and original statements supplemented with solved examples from the same area of physics. Answers were evaluated using numerical results, formulas, and physical units, with numerical accuracy used as the primary performance measure. The original problem statements achieved the highest overall numerical accuracy at 38.7%. Simplifying the statements gave 26.3% accuracy, while adding solved examples resulted in 34.5% accuracy. English-source problems showed higher numerical accuracy than Russian-source problems in all three conditions, but the datasets differed in difficulty, preventing this difference from being attributed to language alone. The results suggest that additional prompt structure or context does not necessarily improve physics problem solving by language models and that preserving information contained in natural textbook-style problem statements may be important. Scripts and data for this experiment are available in https://github.com/BOCK134/Physics_problems_solving_methods.