Beyond Segmentation Accuracy: Evaluating Biological Fidelity in Spatial Pathology

Abstract

Multiplex immunofluorescence (MxIF) resolves cellular identity, state, and spatial organization within tumor tissue, enabling higher-order representations of cancer ecosystems including immune infiltration, multicellular neighborhoods, and tissue niches. Yet these analyses depend on a deceptively simple first step: accurately identifying individual cells. Segmentation errors can alter marker quantification, cellular phenotypes, cell–cell relationships, and ultimately the spatial organization inferred from an image. Consequently, improvements in conventional segmentation benchmarks matter biologically only if they propagate to more faithful representations of the tumor microenvironment. We test this relationship directly using an expert-curated benchmark of 31,924 expert-reviewed cells on three slides under a leave-slide-out protocol. We compare a domain-specific self-supervised encoder, adapted to multiplex tissue imaging, with supervised, microscopy-specific, prompted generalist, and general-purpose vision models, and pass every method’s predicted cells through an identical downstream pipeline. We evaluate fidelity across four levels: cell segmentation, phenotype assignment, spatial neighborhood structure, and tissue organization, comprising niches and tumor–immune interactions. The resulting hierarchy reveals that these levels do not improve in lockstep. Cellular phenotype assignment is comparatively robust to segmentation differences, whereas missed cells increasingly disrupt neighborhood structure and tumor–immune interactions. As a stringent spatial-interaction test, tumor–CD8+ contact recovery is poorly predicted by conventional segmentation quality (Panoptic Quality: ρ = 0.18), while errors specifically at tumor–immune interfaces are strongly associated with contact fidelity (ρ = −0.80). Methods with nearly identical segmentation accuracy can therefore produce substantially different biological representations. These results argue that cell segmentation for spatial pathology should be evaluated not only by boundary and instance accuracy, but also by preservation of the biological relationships that downstream analyses are intended to measure.