This talk presents case studies in which evaluation metrics reward surface-level improvements like polish and fluency while overlooking evidentiary content, correctness, and editorial usefulness, and discusses practical lessons for building reliable evaluation systems in real peer review workflows.
Negar Arabzadeh, Sajad Ebrahimi, Shakiba Amirshahi et al.· Proceedings of the 35th ACM...· 0 citations
The Third Search Futures Workshop [Azzopardi et al., 2026], in conjunction with the Forty-eight European Conference on Information Retrieval (ECIR) 2026, looked into the future of search to ask questions such as: • How can we navigate data privacy in large language model (LLM)-based information retrieval (IR)? • How ca...
Leif Azzopardi, Charles L. A. Clarke, C. Hauff et al.· ACM SIGIR Forum· 1 citation
Findings show that many peer-review evaluation metrics partially conflate review quality with linguistic presentation, and indicate that robustness to meaning-preserving rewriting should be validated before such metrics are used to compare human-written, AI-assisted, and AI-generated reviews.
Shakiba Amirshahi, Sajad Ebrahimi, Hai-Son Le et al.· Proceedings of the 35th ACM...· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.