The comparative analysis of banks'financial statements poses significant challenges for automated question answering systems due to their complexity, substantial length, technical language, and inhomogeneity of both textual and numerical content across different jurisdictions and institutions. We introduce FinRAG-QA, a novel benchmark dataset for financial question answering, which comprises 999 practitioner-curated questions on 10 standardised indicators, grounded in 209 annual and Pillar 3 reports from 24 major European and U.S. banks spanning 2019-2023. Unlike prior financial QA benchmarks, which centre on U.S. filings and single-institution analysis, FinRAG-QA targets cross-institutional retrieval over documents averaging 198k words, longer than any existing financial QA resource. On this benchmark we evaluate a multi-stage RAG pipeline and isolate the contribution of each component. Contextual chunk enrichment combined with a retrieval-optimised embedding model raises NDCG@10 from 0.322 to 0.710; conditional on the ground truth being retrieved, a reasoning-optimised generator raises answer accuracy from 44.6% to 79.0% (+34.4 percentage points), at roughly 20x the generation latency. We further show that cross-encoder reranking degrades retrieval when the first-stage ranking is already strong, and that a single top-ranked chunk outperforms larger contexts at generation time. Experiments were run in late 2024-early 2025 with the models available at that time.
Arianna Miola, Bruno Spaccavento, Lorenzo Silotto et al.· 0 citations
Aspect-based Summarization (AbSum) aims to generate document summaries describing different facets of the content. The summaries produced by a Large Language Model (LLM) fine-tuned for AbSum likely cover all aspects in the training set, even though some of them may be undesired. Machine Unlearning (MU) techniques are designed to remove the influence of specific training samples from trained models. Applying MU to disentangle fine-tuned LLMs from undesired aspects is particularly appealing, as it enables efficient model reuse. Despite their relevance and diffusion, MU techniques have not yet been adapted to AbSum. In this work, we bridge the gap by proposing and addressing a new task, i.e., MU for LLM-based Summarization (MU4Sum, in short). Since a document may include both aspects to forget and aspects to retain, we extend traditional MU methods to operate at the token level. Then, we empirically compare the token- and sample-wise approaches using 3 LLMs and 12 state-of-the-art MU techniques on 4 public datasets suited to MU4Sum. Our approaches perform significantly better than existing techniques on both the retain and forget test sets. Furthermore, we propose to integrate MU4Sum into an extractive-then-abstractive summarization pipeline, where the content discarded by an extractive summarizer is treated as the forget set. MU4Sum improves the performance of the traditional pipeline, highlighting its potential for broader applications.
Giuseppe Gallipoli, Luca Cagliero· ACM Transactions on Intellig...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
