Refining scRNA-Seq Clusters: The Power of Feature Selection

Abstract

Feature selection is critical for resolving cell-type heterogeneity in single-cell RNA sequencing (scRNA-seq). DUBStepR (Determining the Underlying Basis using Stepwise Regression) is a widely used gene selection method for scRNA-seq designed to identify feature genes that maximize cell-type separation. DUBStepR has been reported to perform effectively in this domain; however, its reliance on linear Pearson correlation and rigid thresholding limits its effectiveness on complex, high-dimensional datasets. Three enhancements are presented in this study: RFCell-DUBStepR, which uses random forests to capture expression-level importance; Copula-DUBStepR, which models non-linear correlations via Gaussian Copulas; and Zqt-DUBStepR, which utilizes quantile-based selection for improved gene retention. Using both simulated and real-world datasets (scRNA-seq), these modifications are shown to resolve the biases of the original algorithm. The modified methods consistently select a more representative gene set and yield higher clustering accuracy across varying levels of biological complexity. These findings establish the modified DUBStepR frameworks as more reliable tools for high-fidelity subpopulation identification in downstream single-cell analysis.