A label- free, multi-metric pipeline is presented that reframes clustering as an auditable, methods-blind decision and separates two notions of stability that are commonly conflated: reproducibility under cell resampling (bootstrap) and reproducibility under re-embedding (retraining the representation).
Abstract
A discovered sub-population from single-cell transcriptomic data is only meaningful if it is reproducible, yet clustering is usually done with one method on one embedding and rarely tested. We present a label- free, multi-metric pipeline that reframes clustering as an auditable, methods-blind decision and separates two notions of stability that are commonly conflated: reproducibility under cell resampling (bootstrap) and reproducibility under re-embedding (retraining the representation). The pipeline evaluates seven clustering configurations across cluster counts using five non-redundant quality metrics. As a whole- dataset control on a mouse retinal atlas, it recovers an eight-cell-type annotation at 96.3% accuracy (adjusted Rand index, ARI = 0.91) without labels. We then validate the discovery mode on two cell types with opposite ground truth. On bipolar cells, which have well-established subtypes, the pipeline accepts the sub-structure: across-embedding reproducibility rises with cluster number to a high plateau (mean pairwise ARI ∼0.93 near the ∼15 known bipolar subtypes), with quality metrics improving in parallel. On rod photoreceptors, treated as homogeneous, it rejects over-clustering: the metric-selected partition passes a bootstrap-stability check but is not reproducible when the embedding is retrained (mean pairwise ARI = 0.69), and the metrics do not improve with cluster number. On synthetic data, the test recovers real structure down to a 5% subpopulation while rejecting null data (high sensitivity and specificity). Bootstrap stability alone is therefore insufficient evidence for sub-population; the across-embedding test discriminates real sub-structure from over-clustering and applies to any cell type as a reproducible alternative to single-method, single-embedding clustering.
Automated cell-type annotation is a prerequisite for most single-cell RNA-sequencing (scRNA-seq) analyses, but the rapid proliferation of methods spanning marker-based, similarity-based, classical machine-learning, deep-learning, semi-supervised, large-language-model (LLM), and transformer foundation-model paradigms has outpaced head-to-head evaluation. Existing benchmarks rely on convenience samples of real datasets in which cell count, class imbalance, cell-type number, and differential-expression strength co-vary uncontrollably, precluding causal attribution of performance to any dataset property. To resolve this, we benchmarked 63 tools across seven paradigms using a Taguchi L9(3⁴) orthogonal array that varies four dataset properties independently, progressively reconfiguring experimental control across five phases: fully controlled simulation, within-platform and cross-platform real-data validation, database-connected and LLM-based annotation under ontology-aware scoring, and fine-tuned foundation models. Using standardized oracle inputs and Cohen’s κ, we found that, within the ranges tested, the major paradigms achieved comparable accuracy. Accuracy was predicted near-linearly by the separability of cell types in a shared expression embedding, measured as k-nearest-neighbor (kNN) purity, a relationship that held across sequencing platforms and in fine-tuned foundation models. We attributed the vast majority of κ variance to dataset structure and only a small share to tool identity. Computational cost traded against workflow accessibility rather than accuracy: accessible similarity-based and LLM-based approaches performed competitively, while foundation models matched them only after fine-tuning. Because our oracle design isolates algorithmic capability from upstream noise, these results reframe how methods should be selected: the field’s near-term gains lie in strengthening infrastructure—prioritizing tool accessibility, standardized evaluation, and robustness to pipeline variation.
Oliver Wardhana, Ziyu Zeng, Xin Lu· bioRxiv· 0 citations
Single-cell RNA sequencing has enabled the construction of comprehensive cell atlases, yet the quality and coherence of the cell-type annotations within these atlases remain largely unexamined. When a label is applied to a transcriptionally heterogeneous population, the downstream analyses that depend on it, and automated label transfer in particular, become unreliable. We present scINTILLA (Single-Cell Integrated Inference, Labelling, and Landscape Analysis), a computational framework that combines supervised and unsupervised machine learning to score the learnability and internal consistency of cell-type labels in single-cell datasets. The unsupervised arm benchmarks a broad panel of clustering algorithms and derives a neighbourhood confusion score for every cell, whilst the supervised arm trains up to twelve classifiers and extracts prediction agreement, entropy, and confidence. These signals are normalised and aggregated into a single composite score per cell type, where a low score flags label ambiguity or concealed heterogeneity. As a by-product, scIN-TILLA also reports which clustering and classification algorithms perform best on a given dataset, offering practical guidance for downstream label transfer. We applied it to five Human Cell Atlas datasets spanning the adult brain, lung, eye, and two organoid atlases, and recovered clear differences in the learnability and internal consistency of annotations across atlases that were not driven by the number of annotated cell types. Focused re-analysis of lowscoring populations in the lung and endoderm-organoid atlases resolved biologically coherent sub-populations, in some cases with context-specific enrichment, much of it recovered from cells that had been assigned broad or catch-all labels. scINTILLA is advisory rather than prescriptive, guiding principled, data-driven re-annotation at atlas scale.
Sina Kanannejad, Noemi Bongiorni, Elisa Nordera et al.· bioRxiv· 0 citations
Per-dataset analysis reveals three reproducible regimes: probabilistic variational autoencoder variants help on the smallest datasets, deep autoencoders win on mid-scale data with multi-batch or many-type structure, and classical PCA pipelines remain competitive when linear projection already captures the dominant variation.
Phong T. Nguyen, T. Vu, T. Nguyen et al.· arXiv.org· 0 citations
Single-cell proteomics (SCP) currently lacks validated benchmarking standards, and cell annotation often relies on transcriptomic proxies. Unsupervised clustering offers a proxy-free alternative, but its success depends on biological signal outweighing technical variation. In homogeneous samples this is achievable, but in heterogeneous populations, where closely related cell types differ only subtly, technical variation can dominate the clustering and obscure the biology needed for annotation. To address this, we developed an integrated experimental and computational pipeline for protein-level cell annotation and applied it to human PBMCs as an immune-cell test case. We isolated T cells, B cells, monocytes, and NK cells by negative-selection sorting to build a high-fidelity reference. In parallel, unsorted PBMCs from the same donor were processed on a cellenONE and acquired using label-free DIA on an Orbitrap Astral Zoom. Using the labeled reference dataset, we systematically benchmarked normalization, imputation, and clustering methods to assess their effect on cell-type separation. Unsupervised analysis resolved functional subpopulations within each lineage, and a probabilistic SCP classifier trained on these annotations identified the corresponding cell types and states in the unsorted PBMC fraction, validating the pipeline on unenriched, heterogeneous samples. Together, this work delivers an analytically benchmarked SCP workflow that resolves immune lineage and cell-state heterogeneity in human PBMCs and provides a classifier-ready, protein-level reference for immune-cell assignment.
S. O'Connor, Romell B. Gletten, Ritin Sharma et al.· bioRxiv· 0 citations
This work introduces CytoGate-Bench, a benchmark that reformulates this per-step procedure as a zero-shot, panel-agnostic task for large language models, and contributes a public benchmark that tests precisely that ability across 11 human cohorts.
Jaesik Kim, Byounghan Lee, Namhyuk Ahn et al.· bioRxiv· 0 citations
Malva is presented, a computational platform that enables ultrafast, species-agnostic and reference-free interrogation of the raw sequence space, enabling searching for any sequence, mutation, splice junction or pathogen, or spatial location of arbitrary transcripts.
D. León-Periñán, Nikos Karaiskos, N. Rajewsky· Nature· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
