Error Correction Code Residual Transformer

Abstract

The Error Correction Code Transformer (ECCT) is a seminal state-of-the-art model-free neural decoder for linear block codes, leveraging parity-check-matrix-aware masked self-attention to exploit algebraic code structure. A well-known limitation of deep Transformer architectures trained with Pre-Layer Normalization (PreNorm), however, is the uncontrolled growth of hidden-state magnitudes with depth, which progressively dilutes the contribution of individual layers, a phenomenon known as PreNorm dilution, thus effectively limiting the benefit of increased architectural depth. In this paper, we propose EC-CRT, a novel Transformer-based neural decoder that integrates Attention Residuals (AttnRes) into the ECCT framework to address this limitation. AttnRes replaces fixed, additive residual connections with learned, input-dependent softmax attention over all preceding layer outputs, enabling each decoding layer to selectively aggregate the most informative earlier representations and directly mitigating PreNorm dilution. Extensive experiments on LDPC(49,24), Polar(64,32), BCH(31,16), and BCH(63,51) codes under a deep configuration and a shallow configuration demonstrate that ECCRT consistently and substantially outperforms ECCT in the deep setting across all codes and SNR values, with gains that widen at higher SNRs, while delivering comparable performance in the shallow setting. The improvement is achieved with negligible additional parameters, only one learned pseudo-query vector per Transformer layer. To the best of our knowledge, this is the first work to demonstrate the benefit of Attention Residuals outside the large language model regime, confirming that this technique generalizes across model scales and application domains.