CLINLENS is introduced, a benchmark of 200 executable tasks over five linked MIMIC resources spanning structured electronic health records, notes, electrocardiograms, chest radiographs, and echocardiograms, which exposes a substantial gap between runnable submissions and correct clinical analyses.
Abstract
Clinical data-science agents must transform heterogeneous longitudinal records into auditable analyses, yet existing benchmarks largely isolate medical question answering, structured-table reasoning, or generic scientific repositories. We introduce CLINLENS, a benchmark of 200 executable tasks over five linked MIMIC resources spanning structured electronic health records, notes, electrocardiograms, chest radiographs, and echocardiograms. A 4 x 5 taxonomy crosses four patient-time scopes with five analysis capabilities. Program-first reverse synthesis pairs each bounded semi-raw package with an evaluator-private reference workflow and checks required artifacts, cohort and temporal semantics, and the final answer. On a fixed 126-task suite, the strongest of 24 standardized model-scaffold configurations achieves 56.3% scope-macro STRICTPASS despite 100% EXECSUCCESS. For reference, a separately configured coding agent solves 83 of 126 tasks, while five biomedical systems adapted to GPT-4o-mini reach at most 2.9% scope-macro STRICTPASS. These results expose a substantial gap between runnable submissions and correct clinical analyses.
Laboratory medicine sits at the intersection of clinical science and data management. A single hospital admission can generate dozens of analyte values, yet most electronic health record (EHR) interfaces present them as rows in a table, leaving interpretation entirely to the clinician. Alert fatigue, driven in part by poorly calibrated notifications remains one of the most documented usability problems in modern EHR design [1].
This paper describes a prompt-driven pipeline that converts FHIR R4 laboratory panels into structured, paragraph-length clinical narratives paired with a calibrated 0-1 risk score, using GPT-4o-mini as the generation engine. The full behavioral specification is encoded in the prompt and output schema. We evaluated the system on 200 laboratory panels, each drawn from a distinct synthetic patient, from Synthea-generated FHIR bundles spanning seven panel categories (metabolic, lipid, blood count, diabetes monitoring, kidney, liver, and urine). We compared three configurations: a rule-only template baseline, the LLM alone (no seed), and the hybrid pipeline in which a deterministic rule-based risk seed is supplied to the LLM. Abnormal-analyte detection was near ceiling and statistically indistinguishable for both LLM configurations (F1 ≈ 0.97), indicating that the model recovers out-of-range analytes directly from the structured table with or without the seed. The seed's measurable contribution is to risk-score calibration: the correlation between the model's 0–1 risk score and the reference rule score rose from r = 0.87 (no seed) to r = 0.96 (with seed). The rule seed thus functions as a calibration mechanism rather than a detection aid.
These are proof-of-concept results on synthetic structured data. They demonstrate feasibility for abnormality flagging and narrative generation; they do not constitute a claim of clinical validity, which would require real-world data and clinician review. The paper contributes a reproducible architectural framework, a systematic quantitative benchmark on synthetic panels, and a grounded discussion of the integration challenges and future directions that separate a research prototype from a clinically deployed tool.
Rahul Reddy Hanumanthgari· EPJ Web of Conferences· 0 citations
MOSAIC shows agentic LLM systems can generate and apply clinically meaningful severity phenotypes from structured EHR data in T2D, and extends it to other diseases with similarly multidimensional severity warrants further research.
Manuela Del Castillo Suero, Arnault-Quentin Vermillet, Nicole Sonne Heckmann et al.· arXiv.org· 0 citations
CardioBench is the largest real-world, multi-task benchmark for LLM evaluation across the cardiovascular care continuum and offers the broadest coverage of clinically authentic cardiology scenarios reported to date.
Xiao Li, Mouxiao Bian, Zhaodi Wu et al.· 0 citations
In this work, we introduce LongMedBench, a real-world EHR-based benchmark for long-horizon clinical decision-making. Prior evaluations of LLM-based medical agents have largely emphasized short-context knowledge QA and tool use. However, real-world medical care is inherently longitudinal, and clinicians must aggregate evidence across repeated visits, tests, and evolving treatments. Therefore, long-horizon interaction is essential for realistic assessment. LongMedBench is constructed via a reproducible pipeline that integrates MIMIC-IV admission records and clinical notes into time-series event streams and long-context memory datasets, enabling long-horizon, multi-session interactions between agents and a clinical environment. It comprises 335 patients, with 19.72 inpatient visits per patient on average and 44.91 medical events per visit. Guided by the long-horizon decision process, we propose an evaluation taxonomy with three suites: fact-based QA, temporal reasoning, and long-horizon decision-making. This taxonomy measures how agents understand and leverage historical patient information over extended horizons. Our experiments show that while recent LLMs can make good use of explicit timestamps, they have challenges in implicit time inference; The RAG and agent memory system can improve the performance of information retrieval tasks, but the performance of decision-making tasks is highly dependent on the model's immediate context.
Zihan Xu, Yanzhen Chen, Xiaochen Zhang et al.· 0 citations
Objective: Health systems and biopharma face a gap between holding patient data and acting on it: records are fragmented, manually mapped, and structured for storage rather than decisions, so every application re-derives patient state. We present PRomop, an open-source longitudinal record that closes this gap. Materials and Methods: PRomop builds on the OMOP Common Data Model (CDM 5.4) with oncology extensions and adds PatientRecord, a flattened projection collapsing each patient's longitudinal history into a single decision-ready 304-column row. State derivations - lines of therapy, disease status, normalized biomarkers - are computed once at projection time, so analytics, trial matching, and standard-of-care evaluation read one substrate. Results: PRomop is deployed by two oncology organizations - the independently governed HealthTree Foundation (~14,000 patients) and CancerBot (~3,500), a HealthKey-owned deployment - matching against 19,500 recruiting trials across five cancer types. A 20-criterion eligibility search requiring 27-39 joins over raw OMOP reduces to zero against the projection. On a synthetic 1000-patient breast-cancer cohort, eligibility screening averaged 0.30 ms via PatientRecord versus 11.0 ms from raw OMOP, a ~36.8x speedup. Discussion: The projection's significance is as a foundation for other applications: it lowers each one's marginal cost by computing error-prone clinical derivation once and removing it from every consumer. Line-of-therapy inference showed decision-readiness demands embedded clinical reasoning, and that the projection is a living artifact requiring maintenance. Conclusion: A flattened, decision-ready projection over a standards-based longitudinal record is a deployed pattern for turning fragmented data into actionable infrastructure, while remaining OMOP-conformant. Benchmarks measured a ~36.8x eligibility-screening speedup.
A. Blum, Louis Ferger-Andrews, Steven Labkoff· arXiv.org· 0 citations
CLIR-Bench is introduced, a benchmark for irregular clinical time series QA constructed from de-identified ICU records through a principled four-stage pipeline, enabling evaluation of both answer accuracy and evidence use.
Frank Nie, Ethan B. Liu, Yuan Zhu et al.· arXiv.org· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
