A systematic evaluation framework for BIM-QA systems comprising a four-category question taxonomy based on data retrieval complexity, and an evaluation protocol with five binary quality criteria, enabling consistent benchmarking and targeted diagnosis of system weaknesses is presented.
Eagle, a comprehensive benchmarking framework tailored for evaluating OpsLLMs, delivers a deployable foundation for advancing large-model applications in AIOps and open-source the framework and dataset to foster community adoption and reproducibility.
Yuhe Liu, Changhua Pei, Hang Wang et al.· SIGSOFT FSE Companion· 0 citations
This work proposes a comprehensive pipeline for improving financial QA systems through high-quality synthetic data generation and fine-tuning of smaller language models (SLMs) using Quantized Low-Rank Adaptation (QLoRA).
Lokendra Birla, Milind Savagaonkar, Visnu Srinivasan et al.· 0 citations
CoAL-RAG is proposed, a complexity-aware legal retrieval-augmented generation method, which constructs a multi-dimensional evaluation mechanism based on ``question essence'' and ``retrieval consistency'' to enable adaptive routing of retrieval strategies.
Jinyue Su, Zhuofeng Zhao, Huanhuan Wang et al.· 2 citations
Three novel metric models are proposed: a prompt-based strategy utilizing Large Language Models to assess answers, an approach that adapts precision and recall concepts by segmenting answers into discrete information units, and a regression model trained on synthetic data to predict completeness and relevance scores.
E. Cortes, Renata Vieira, D. Barone· Language Resources and Evalu...· 0 citations
The results demonstrate that lightweight, task-specific reasoning strategies can provide strong performance on resource-constrained multimodal question answering benchmarks.
This study develops a multi-source Retrieval-Augmented Generation (RAG) based Question Answering (QA) system that automatically integrates heterogeneous knowledge sources through a unified source parameter to enhance knowledge transfer and question answering for organizational support and employee onboarding.
Krisna Dwi Setya Adi, Ivan Michael Siregar· Jurnal Ragam Pengabdian· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
