Cell-type separability predicts annotation accuracy and outweighs algorithm choice: a factorial benchmark across seven scRNA paradigms

Abstract

Automated cell-type annotation is a prerequisite for most single-cell RNA-sequencing (scRNA-seq) analyses, but the rapid proliferation of methods spanning marker-based, similarity-based, classical machine-learning, deep-learning, semi-supervised, large-language-model (LLM), and transformer foundation-model paradigms has outpaced head-to-head evaluation. Existing benchmarks rely on convenience samples of real datasets in which cell count, class imbalance, cell-type number, and differential-expression strength co-vary uncontrollably, precluding causal attribution of performance to any dataset property. To resolve this, we benchmarked 63 tools across seven paradigms using a Taguchi L9(3⁴) orthogonal array that varies four dataset properties independently, progressively reconfiguring experimental control across five phases: fully controlled simulation, within-platform and cross-platform real-data validation, database-connected and LLM-based annotation under ontology-aware scoring, and fine-tuned foundation models. Using standardized oracle inputs and Cohen’s κ, we found that, within the ranges tested, the major paradigms achieved comparable accuracy. Accuracy was predicted near-linearly by the separability of cell types in a shared expression embedding, measured as k-nearest-neighbor (kNN) purity, a relationship that held across sequencing platforms and in fine-tuned foundation models. We attributed the vast majority of κ variance to dataset structure and only a small share to tool identity. Computational cost traded against workflow accessibility rather than accuracy: accessible similarity-based and LLM-based approaches performed competitively, while foundation models matched them only after fine-tuning. Because our oracle design isolates algorithmic capability from upstream noise, these results reframe how methods should be selected: the field’s near-term gains lie in strengthening infrastructure—prioritizing tool accessibility, standardized evaluation, and robustness to pipeline variation.