2026· IEEE Transactions on Learning Technologies· Vol 19, pp. 752-765· 0 citations· 48 references
Abstract
Agentic tutoring systems increasingly rely on large language models (LLMs) to generate real-time feedback, yet risky outputs, such as answer leakage, contradiction, and hallucination, can directly affect learning. Answer leakage is a central concern in this setting. It exposes problem solutions to students and can be verified through human annotation. This article presents a pedagogical style-conditioned audit protocol for automated evaluators that detect answer leakage in LLM-generated tutoring feedback before deployment. Using 1100 mathematics problems from the MATH dataset and 4400 feedback items generated under Baseline, Direct, Socratic, and Encouragement styles, we compare rule-based, embedding-based, natural language inference-based, and two LLM-as-a-Judge evaluators from different model families (Qwen2.5-32B and Gemma3-27B) and validate them against a 314-item human-annotated gold standard. The results show that pedagogical style changes how answer-leakage signals appear in feedback and how reliably automated evaluators detect them. Rule-based and LLM-Judge evaluators that perform well under Baseline, Direct, and Encouragement show consistent degradation under Socratic feedback. The primary LLM-Judge (Qwen2.5-32B) achieves overall F1 = 0.726 but drops to 0.427 on Socratic items, and a second LLM-Judge (Gemma3-27B) reproduces this degradation with Socratic F1 = 0.342. The four factorial prompt-sensitivity variants reach overall F1 up to 0.844, but their Socratic F1 remains at or below 0.390. These findings show that automated tutoring-feedback auditing must be style-specific, prompt-calibrated, and human-grounded before deployment.
A post hoc robustness audit of the same 1,179 confirmatory answer-phase tutor turns under the frozen helpfulness and pedagogy rubrics finds that general-purpose helpfulness is not a reliable pedagogy signal.
FATE (FLC AI Tutor Evaluator), a specialized 8B-parameter language model designed to evaluate AI tutors, is introduced, which assesses pedagogical ability across Mistake Identification, Mistake Location, Guidance, and Actionability.
Tahmid Al Hannan, Diego García, Alex K Njoroge et al.· 0 citations
These results establish an auditable release boundary and failure attribution under declared contracts, not universal semantic safety or learning gains, not universal semantic safety or learning gains.
Textual skills provide a lightweight way to improve frozen language-model agents, but their self-evolution normally requires a stable validation signal. Such signals are natural in mathematics or code, where an answer can be checked after it changes, yet are problematic in open-ended dialogue: changing the assistant response also changes the user's next reaction, so a logged reaction cannot directly evaluate a counterfactual response. We propose future-feedback skill evolution, which first redirects self-evolution from prescribing the current answer to predicting whether the observed answer will lead to a positive or negative subsequent user signal. This prediction task is verifiable on fixed logged tuples and therefore supports validation-gated textual optimization. The evolved feedback skill captures interpretable criteria for response quality and can subsequently serve as a diagnostic and optimization target for answer skills. On a proprietary, privacy-preserving sales-assistant dataset, careful quality filtering and a balanced resolved/unresolved split yield more than 75% prediction accuracy. Beyond this result, the central contribution is a formulation that converts otherwise moving conversational feedback into a fixed offline learning target, enabling reproducible skill evolution without placing every candidate skill in live traffic. We discuss the boundary between observational verification and counterfactual validity, and position the method as an offline optimization stage rather than a replacement for final human or online evaluation.
Current K12 online math tutoring platforms share a common weakness: they tend to identify student errors only at a surface level, and when it comes to feedback, most systems simply hand out the correct answer. Over time, this combination chips away at the pedagogical promise of intelligent tutoring. The present work tackles both problems through a lightweight diagnostic-feedback pipeline that pairs MathBERT for semantic analysis with SymPy for symbolic verification. Why two engines? Because MathBERT reads the meaning behind a student’s response — catching conceptual slips that pattern-matching would miss — while SymPy checks whether two mathematical expressions are genuinely equivalent, which eliminates the well-documented hallucination tendencies of language models in algebraic contexts. For the feedback component, parameter-efficient fine-tuning (PEFT) first injects math-domain knowledge into a compact language model, after which direct preference optimization (DPO) steers the model toward Socratic-style questioning instead of answer-revealing behavior. Tested against the MathDial corpus, the pipeline reaches 99.17% on answer verification accuracy, posts an F1 of 0.9139 for error localization, and keeps the direct-answer rate down to 4.2% — beating every baseline on all three counts. These numbers suggest that symbolic checking paired with preference alignment offers a practical path toward tutoring systems that are both reliable and genuinely instructive, even on modest hardware.
Xuchen Lin· ITM Web of Conferences· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
