Beyond Topic Matching: Goal-Conditioned Evaluation of Research Contributions in ETD Collections
TL;DR
The loss of fidelity shows that intermediate teacher evaluations provide useful supervision for approximating the evaluative function, and whether reducing the influence of intermediate teacher evaluations improves or impairs approximation is tested by comparing plain binary cross-entropy with weighting, masking, and hard-label objectives.
Abstract
Evaluating whether scholarly research contributes to a Sustainable Development Goal (SDG) requires an interpretive framework that combines the goal definition, its targets, contribution criteria, and the document itself. Instruction-tuned large language models can instantiate this framework in context, producing a consistent evaluative function that can be applied to every document in a collection. We present Goal-Conditioned Multi-Label Distillation (GCMLD), which distills this evaluative function into a compact multi-label model. A Qwen2.5-7B-Instruct teacher evaluates each document–SDG pair by constraining its generative output to two tokens ("0" and "1"), and the logit margin defines the teacher probability used to supervise the student. An 86-million-parameter DeBERTa-v3-base student learns to approximate the teacher’s evaluative function by predicting all seventeen SDG scores in a single model invocation. We test whether reducing the influence of intermediate teacher evaluations improves or impairs approximation by comparing plain binary cross-entropy with weighting, masking, and hard-label objectives. Masking small-margin evaluations reduced teacher fidelity with the teacher targets and temperature held fixed, and the reduction was largest for SDG–document pairs whose own margins are small. The loss of fidelity shows that intermediate teacher evaluations provide useful supervision for approximating the evaluative function. The student trained on publication abstracts continued to approximate the teacher on electronic thesis and dissertation (ETD) abstracts, with ETD-specific distillation producing modest additional gains. Replacing seventeen prompted teacher evaluations with one model invocation makes collection-scale evaluation practical. Aggregating document-level contribution probabilities by academic department demonstrates how the distilled model supports collection-scale analysis of institutional research.
