Predicting Scholarly Impact with Temporal Preference Alignment
TL;DR
Impact-DPO integrates temporally informed prompting with direct preference optimization, enabling LLMs to learn comparative influence patterns without explicit graph message passing, and formalizes citation forecasting as pairwise preference learning on temporal text-attributed graphs.
Abstract
Background. Predicting the future influence of scientific papers remains a longstanding challenge in bibliometrics and information retrieval. Traditional regression and embedding-based methods estimate citation counts from textual features but fail to capture the inherently relational and temporal nature of scholarly impact. Objective. This paper proposes a preference-aligned framework for forecasting and generating scholarly influence from paper abstracts and temporal cues. Rather than predicting absolute citation values, we model the relative likelihood that one paper will accrue more citations than another within a shared temporal time frame. Methodology. Impact-DPO integrates temporally informed prompting with direct preference optimization, enabling LLMs to learn comparative influence patterns without explicit graph message passing. We formalize citation forecasting as pairwise preference learning on temporal text-attributed graphs, using publication year as a minimal temporal signal. Experiments were conducted on two large-scale domains, i.e., Computer Science and Physics, spanning three temporal splits (2018–2020). Results. We show that Impact-DPO achieves the highest pairwise accuracy across all splits, with an average pairwise accuracy of 83% (up to \({\approx}88\%\) on individual splits), outperforming SPECTER2+SVR by 12 pp and zero-shot prompting by more than 15 pp. Preference alignment yields an average improvement of roughly +30 pp over binary-classification baselines overall; importantly, supplementary same-backbone Qwen 2.5–7B BC runs remain near chance (about 50–58% across splits), showing that the gain is not explained by backbone capacity alone. A lower regularization parameter ( \(\beta=0.1\) ) consistently produces optimal results, consistent with a low-gap preference regime in citation data. Generative evaluations further reveal that model-generated text aligns more closely with highly cited papers (KS=0.0744, p=0.0065), demonstrating emergent generative alignment with influential scholarly language. Code and data availability. All code, data-preprocessing scripts, and evaluation notebooks are available at https://github.com/parhamouni/impact-dpo.