AS-Split Conformer: A Stage-Wise Convolution–Attention Framework with Mamba Decoder for End-to-End Speech Recognition

Abstract

Automatic speech recognition (ASR) systems based on Conformer architectures achieve strong performance by jointly modeling local acoustic patterns and global contextual dependencies. However, their interleaved convolution–attention design leads to progressive entanglement of fine-grained acoustic features and global semantic representations, which weakens monotonic alignment in speech recognition and degrades performance in long utterances. To address this limitation, we propose an AS-Split Conformer–Mamba framework that decouples local and global modeling into two explicit stages. First, a stage-wise encoder is introduced, where a dedicated local modeling stage extracts phonetic-level acoustic features using SE-enhanced convolution, followed by a global modeling stage that captures long-range dependencies via multi-head self-attention and temporal convolution. Second, a Transition Fusion Block (TFB) is designed as an adaptive transition module that transforms local acoustic representations before they enter the global modeling stage. Third, intermediate CTC supervision is introduced to explicitly strengthen monotonic alignment at shallow representations. Finally, a hybrid Transformer–Mamba decoder is adopted, in which the Mamba block provides O(N) state-space computation within the replaced FFN sublayer while retaining Transformer attention mechanisms for acoustic–text alignment. Experiments conducted on AISHELL-1, THCHS-30, and ST-CMDS demonstrate that the proposed method achieves consistent improvements over strong baselines. On AISHELL-1, our model reduces Character Error Rate (CER) from 5.7% to 4.8% and Sentence Error Rate (SER) from 24.8% to 20.5%, while maintaining competitive computational efficiency.