Detection of extremely rare dead cells in fluorescence microscopy images

Abstract

Dead cell detection in fluorescence microscopy is essential for phenotypic screening and drug efficacy assessment. However, automated identification remains challenging due to signal variability, optical artifacts, and colocalization ambiguities across multiple channels. Markers that define nuclear boundaries do not neatly coincide spatially or temporally with apoptotic markers. Furthermore, marker accumulation kinetics vary across individual cells and apoptosis stages, making universal thresholds unreliable. We present a systematic comparison of several computational approaches for detecting apoptotic cells in nucleus-stained specimens: (1) heuristic methods using intensity thresholds, distributions, correlations, and domain guidance; (2) classification using numerical features (Logistic Regression, Random Forest, XGBoost and LightGBM) and image features (ResNet-18 and EfficientNet-B0); and (3) anomaly detection, also using numerical features (Isolation Forest, one-class SVM and PCA reconstruction) and image features (convolutional autoencoders and Deep SVDD). Using a dataset of 258 images containing 65,527 cells/nuclei across multiple scenes with 243 manually annotated ground truth dead cells, we evaluate each method on metrics such as PR-AUC, Precision, Recall, F1-score, and macro-F1. Our results indicate that classification using image features (deep learning) and numerical features (traditional) achieved the best performance, with ResNet-18 and XGBoost significantly outperforming manually designed heuristics and anomaly detection methods. Our findings demonstrate that such biomedical problems remain highly challenging to solve even with the state-of-the-art computational and learning techniques. We aim to provide an effective decision-support tool to researchers while tackling such problems either fully-automatically or in a human-in-the-loop setting as an aide to domain scientists.