Frame-Adaptive Layer Aggregation of Pre-Trained Speech Encoders for Speaker Verification
Abstract
Speaker verification (SV) has recently benefited from large-scale self-supervised learning (SSL) speech Transformers, serving as powerful feature extractors. However, existing systems rely on a static, scalar-based weighted fusion to handle multi-layer representations from SSL encoders, which suffer from biased layer utilization. Further, given high-dimensional feature maps from SSL encoders, utilizing a heavy off-the-shelf SV downstream model struggles with computational bottlenecks. In this work, we propose a lightweight and convolution-free backend that adaptively aggregates layer-wise information on each frame input through a squeeze–excitation attention mechanism, and applies channel- and context-dependent attentive statistic pooling for temporal aggregation. We validated the proposed speaker extractor using various SSL models, and it achieved state-of-the-art performance on VoxCeleb and CN-Celeb benchmarks: 0.37% on Vox-O and an unprecedented 5.51% on CN-Celeb evaluation. These results demonstrate that our lightweight attention pooling-based backend not only effectively extracts speaker-distinguishable cues but also enhances robustness to linguistic domain shifts.