TL;DR
This approach combines classical retrieval methods with the usage of multiple large language model (LLM) agents to generate concise, evidence-based reports to align with DRAGUN’s goal of supporting critical engagement with news.
Large language models answering questions over multi-page documents are expected to cite the supporting pages, yet supplied citations are sometimes inaccurate, and current evaluations score citations at generation time or against text passages: no existing benchmark evaluates whether a system can verify and correct a page-level citation already attached to an answer. We propose AtomCite, an agentic framework that parses an answer into claims, checks each claim against the image of its cited page, and applies a deterministic repair policy. To evaluate it, we introduce DocCite, to our knowledge the first benchmark for systems that verify and correct page-level citations in document images. Built on MP-DocVQA and DUDE, it combines 928 validated injected instances with 2,468 candidate natural errors harvested from frontier- and efficiency-tier models, of which a two-annotator audit confirms 1,909 as genuine errors. Primary labels are assigned deterministically, not by LLM judges, with the human audit as a separate validation layer. Across three model families (Gemini, Claude, and GPT), AtomCite reaches around 93% binary verification accuracy on the injected benchmark, significantly outperforming every OCR-only condition, including a compute-matched control, and exceeding every prior text-based baseline given the same OCR text. Its repair policy lifts citation precision on the injected mix from a constructed 34% to 87-90% while retaining over 90% of correct claims. AtomCite also transfers: with frozen prompts and zero training, it raises the hallucination-detection scores of two open 7-8B models on five public benchmarks above the same models prompted as direct judges. Finally, the audit shows that noise in automatic labels biases measured verifier accuracy and can reverse system rankings, so evaluations relying only on synthetic or automatic labels risk mismeasuring verification capability.
Citation analysis has traditionally been organized around citation intent, function, stance, citation context analysis, and cited text span identification. Recent large language models (LLMs) have been applied to these tasks through prompting, in-context learning, fine-tuning, annotation assistance, and retrieval-augmented generation. Yet it remains unclear whether these studies also develop a more explicit account of citation recognition. This survey reviews LLM-based citation analysis through a recognition-centered lens. We define citation recognition as the identification of the textual and semantic units needed for citation understanding, especially citation context, citation content, and cited content. We summarize pre-LLM assumptions about fixed context windows, interpretative labels, and cited text span grounding, and then synthesize five LLM-based method paradigms: prompting and fine-tuning, LLM-assisted annotation, long-context recognition, retrieval-augmented grounding, and multi-dimensional frameworks. We argue that the next stage of LLM-based citation recognition should give closer attention to boundary quality, content consistency, evidence faithfulness, and reproducibility.
Scientific discovery depends on finding prior literature that shapes what comes next. Existing retrieval systems optimize for relevance and popularity, often favoring central papers over less familiar works that later prove generative. We introduce \textbf{MUSES}, a million-instance benchmark for prospective intellectual-roots retrieval over a fixed 2.33M-paper corpus, with roughly 140K test instances per familiarity tier. To our knowledge, it is the first prospective benchmark at this scale with a shared retrieval task and author-confirmed paper-level root labels. Alongside it, \textbf{CiteRoots} pairs a scalable rhetorical layer over local citation text (LLM judge $\kappa = 0.896$ versus human gold) with a paper-level author-endorsed layer ($n = 1{,}518$ generative-inspiration pairs from 753 focal papers). MUSES organizes difficulty along two axes: a \emph{familiarity} axis spanning CiteNext, CiteNew, and CiteNew-Isolated, and a \emph{functional} axis spanning broad citations, rhetorical roots, and author-endorsed roots. Across 9 method classes, a lean multi-centroid retriever built on SPECTER2 is strongest. Hit@100 falls from 0.534 on CiteNext to 0.424 on CiteNew, 0.205 on rhetorical CiteNew, and 0.171 on author-endorsed CiteNew, a $3.1\times$ decline. In a registered eight-lens full-test audit, roughly half of broad-tier test instances remain unsolved at K=1{,}000. Rhetorical role and author endorsement are distinct: the same judge agrees with endorsement at $\kappa = 0.037$. We release MUSES, both CiteRoots layers, and a distilled open companion judge for future work on prospective retrieval and intellectual roots.
This study presents a clear and reliable framework for classifying the intent behind scientific citations. It combines multi-model reasoning with concepts from social choice theory. Instead of using a single model, this framework employs three open Large Language Models Gemma, LLaMA, and Mistral. Additionally, we combine their ranked outputs using an exponentially weighted Borda method. By doing so, this approach increases agreement among high-confidence predictions, maintains ranking information, and produces stable, high-quality supervision signals. Consequently, it boosts reliability while remaining transparent. To create a strong experimental basis, we built a large, balanced dataset from the UnarXive corpus, which contains structured full-text scientific articles and citation networks. First, we automatically pulled citation contexts and organized them within a DuckDB-based analytical setup. Then, we rebalanced the dataset across rhetorical categories to enhance representativeness and minimize bias. Finally, we categorized each citation context into one of five roles: background, methodology, comparison, extension, or critique. As a result, the resulting dataset provides a robust foundation for training and evaluation. We trained a SciBERT classifier using these ensemble-generated annotations and tested it on a five-category citation intent classification task. The model achieved a macro F1-score of 0.83, an outstanding result for this type of classification. Indeed, this level of performance shows strong reliability given how challenging it is to differentiate closely related citation functions. Moreover, it demonstrates that combining multiple models produces valuable and distinct supervision signals, capturing subtle rhetorical and semantic patterns that single models often overlook. Furthermore, the framework enhances interpretability. Specifically, the explicit weighting system clarifies how each model contributes to the final outcome. In addition, the deterministic tie-breaking method ensures the outputs are consistent and reproducible. Taken together, these design choices maintain explainability without sacrificing effectiveness.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
