OmniSEG: A Multi-Model Ensemble Framework for Retinal Layer Segmentation in OCT Images

Abstract

Automated Accurate segmentation of retinal layers from optical coherence tomography (OCT) volumes is a prerequisite for the quantitative assessment of neurodegenerative and macular diseases. Yet the task is complicated by low interlayer contrast, imaging noise, and wide anatomical variability across patients. In this work we introduce OmniSEG, an ensemble framework that combines a nested UNet with dense skip connections (UNet++), a Swin Transformer-based UNet (SwinUNet), and an attention-guided surface auto-encoder into a unified training pipeline. The model is trained on the publicly available GOALS-2022 dataset using a compound loss that mixes focal cross-entropy, Tversky, gradient-based boundary, and surface smoothness terms while remaining numerically stable under automatic mixed precision (AMP). After 200 epochs the ensemble reaches a mean Dice of 0.864 across six foreground retinal classes, with a surface RMSE decreasing from 122 pixels to 76.5 pixels. Per-class analysis reveals that the OPL class benefits earliest from training (Dice > 0.10 at epoch 197), which reflects early-stage learning on a relatively wider layer rather than a final result, consistent with its wider span and higher contrast relative to thinner layers such as RNFL. These results establish a reproducible baseline for multi-class OCT segmentation and identify concrete directions for future improvement.