Skip to content

BioEVAL: A global, multi-institutional benchmark of large language and multimodal models for bioengineering

Sep 2026 · arXiv.org · Vol abs/2609.30489 · 0 citations · 34 references
Computer Science

TL;DR

Diverse cloud-scale foundation/multimodal models and locally deployable models suitable for inference on consumer-grade GPUs are evaluated, achieving the highest accuracy of up to 90% on MCQs, similarity score of 0.72 on literature synthesis, and accuracy of 80% on a small sample of multimodal reasoning questions, with substantial performance variation across subfields.

Abstract

Large Language Models (LLMs) have demonstrated historic breakthroughs in general reasoning with early successes in biomedical science. However, existing LLM benchmarking emphasizes factual recall, offering limited insight into model performance on frontier and multimodal tasks. We assembled BioEVAL (BioEngineering Validation of AI and LLMs), a global, multi-institutional initiative designed to assess experimental reasoning capability across bioengineering (BE) subfields. BioEVAL spans 11 major BE subfields plus a set of uncategorized items, bringing together 22 research groups to create a PhD-level benchmark comprising 608 evaluation items: 1) 380 multiple-choice questions (MCQs, 359 retained after audit), 2) 218 literature synthesis tasks, and 3) 10 multimodal problems with experimental image interpretation. Benchmark items underwent authoring-group expert review and centralized quality control before evaluation. Following evaluation, a blinded cross-group consensus audit of the highest- and lowest-accuracy MCQ items flagged 21 questions for revision or removal; these were withheld, and all reported MCQ results are computed on the 359 retained items. We evaluated diverse cloud-scale foundation/multimodal models (e.g., ChatGPT, Gemini, and Grok) and locally deployable models suitable for inference on consumer-grade GPUs. Models achieved the highest accuracy of up to 90% on MCQs, similarity score of 0.72 on literature synthesis, and accuracy of 80% on a small sample of multimodal reasoning questions, with substantial performance variation across subfields. Leaderboard rankings characterize current capabilities, limitations, and development priorities across the evaluated BE task categories. BioEVAL is maintained as an extensible benchmark with standardized protocols for continuing expert item contribution and model evaluation.

View source

Similar papers

Review Open access Sep 2026

Evaluating Large Language Models in Clinical Audiology (AUDIOLOGYBENCH): Benchmark Development and Validation Study

Current frontier LLMs show strong recommendation generation but substantial limitations in audiometric numerical interpretation that are shared across models and that an automated adjudicator partially miscalibrated at the per-item level.

Lin-Kai Li, Chang-Geng Mo, Hao-Shuai Zhou et al. · 1 citation
Preprint Aug 2026

MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis

This work introduces MedReaMM, a benchmark specifically designed to evaluate models'ability to synthesize heterogeneous clinical evidence consisting of detailed patient histories alongside multiple medical images into accurate differential diagnoses under a complete-information paradigm.

Lai Wei, Yu-Chao Chen, Zhenbiao Cao et al. · 0 citations
#large language models Review Open access Sep 2026

Generative large language models in medicine: a scoping review of recent methodological advances

By tracing the evolutionary trajectories of these methodologies, this scoping review provides a mechanism-centered framework to inform responsible model development and deployment in medical settings, tailored to task complexity, data characteristics, and resource constraints.

Fang Li, Jian-Fu Li, Weiguo Cao et al. · 0 citations
#artificial intelligence Preprint Sep 2026

MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain

MMTClinic is presented, a benchmark designed to evaluate large language models (LLMs) on complex reasoning and question-answering tasks involving clinical time-series and reveals notable differences in model performance across tasks, languages, and modalities, highlighting current limitations in clinical reasoning capa...

Sourav Malakar, Harshit Nigam, Akash Ghosh et al. · 0 citations
Preprint Aug 2026

A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench

VITA's advantages in accuracy and completeness persisted under the neutral judge; its communication scores were lower, and this results indicate that a purpose-built clinical RAG system remains competitive with frontier LLMs on an open benchmark, consistent with corpus specificity as a design variable that improves gro...

P. Reddy, C. Mandke, Suvrankar Datta et al. · 0 citations
#artificial intelligence Preprint Aug 2026

SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs'Robustness to Contradictory Evidence

SUP-MIMIC is proposed, a multi-task framework utilizing MIMIC-IV-v3.1 that comprises Basic Assessment, Diagnostic Divergence Task (DDT), and Diagnostic Convergence Task (DCT), designed to evaluate the model's"one-to-many"disambiguation capability among phenotypically similar cases, while DCT assesses the model's abili...

Yi Yu, Bo Wang, Chong Feng et al. · 0 citations

Related blog posts

Hugging Face Blog Nov 10, 2026

Decision Models in Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.