Skip to content
Open access

Frame-Adaptive Layer Aggregation of Pre-Trained Speech Encoders for Speaker Verification

2026 · IEEE Signal Processing Letters · Vol 33, pp. 3921-3925 · 0 citations · 35 references

Abstract

Speaker verification (SV) has recently benefited from large-scale self-supervised learning (SSL) speech Transformers, serving as powerful feature extractors. However, existing systems rely on a static, scalar-based weighted fusion to handle multi-layer representations from SSL encoders, which suffer from biased layer utilization. Further, given high-dimensional feature maps from SSL encoders, utilizing a heavy off-the-shelf SV downstream model struggles with computational bottlenecks. In this work, we propose a lightweight and convolution-free backend that adaptively aggregates layer-wise information on each frame input through a squeeze–excitation attention mechanism, and applies channel- and context-dependent attentive statistic pooling for temporal aggregation. We validated the proposed speaker extractor using various SSL models, and it achieved state-of-the-art performance on VoxCeleb and CN-Celeb benchmarks: 0.37% on Vox-O and an unprecedented 5.51% on CN-Celeb evaluation. These results demonstrate that our lightweight attention pooling-based backend not only effectively extracts speaker-distinguishable cues but also enhances robustness to linguistic domain shifts.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.