67 Foundation Model Embeddings Identify Spatial Restructuring Associated with Immunotherapy Treatment Derived from Hemoxylin and Eosin Images

Abstract

Abstract Background Patients with clear cell renal cell carcinoma (ccRCC) who receive immunotherapy in the first line setting commonly respond to treatments yet the majority develop resistance. A minority of patients will present with primary resistance to immunotherapy, making predicting patients’ response to immunotherapy a critical milestone in deciding appropriate treatment plans given the expanding list of options available. Hemoxylin and Eosin (H&E) images are frequently obtained through clinical care and contain spatial and morphological characteristics beyond cancer diagnosis. Further, there are many foundation models such as Microsoft’s GigaPath that have been trained on 1.3 billion H&E patches to learn the tissue architecture. Using H&E imaging and AI-based GigaPath, we identified associations between continuous spatial embedding of patient tissue images and exposure to immunotherapy (IO). Methods We generated spatial embeddings from H&E images from 86 matched stroma and tumor cores from 25 ccRCC patients using tissue microarrays (Soupir et al) using GigaPath. A custom AI inference approach was used to increase spatial context of the embeddings. Principle component analysis (PCA) of the embeddings (1536 dimensions) was used for downstream analysis. Individual cores were summarized as mean PCA scores of embeddings and were tested for associations with sample/clinical features. Tumor and stroma (tissue source) were compared before exposure to IO among 8 patients, then immunotherapy exposure (before and after being exposed) was compared within tumor and stroma (14 patients). Wilcoxon rank sum was used to compare aggregate scores between groups. Results Across the 86 TMA cores, 2.03 million embeddings were generated. PCA of the embeddings demonstrated that 25.4% of the variation in the embeddings can be explained by just 10 PCs. PC1 (explaining 11% of variance) represented the tissue/glass interface (or artifact) and was used to remove non-tissue-related embeddings (PC1>5 threshold). Of the first 10 PCs, 4 were significantly associated with the tumor/stroma pathologist annotation (PC2, 4, 5, and 7; p-value = 0.0007 to 0.0047). Across PCs 2, 4, 5, and 7, extremely high or low scores overlap regions of malignant cells. PCA-based embeddings within stroma cores from patient tumors before and after IO exposure showed significant differences within the PC9 feature (p = 0.005), strikingly, this difference was not observed in tumor cores (p = 0.931; Figure 1). Visually, stroma naïve to IO show increased structure of extreme scores in PC9 which overlap with connective tissue or collagen. Conclusions Foundation model embeddings identified significant differences between the stroma among ccRCC patient treated with first line IO regimens which is complementary to our previous findings with single-cell spatial transcriptomics. Interestingly, these preliminary findings suggest that general-purpose models from whole slide images, focusing primarily in the stroma, can be used to extract tissue characteristics that may be otherwise unrealized from H&E images. Further research is needed for external validation and to explore the full potential of the embedding space.