HIFN-Transformer: Learnable Information-Theoretic Parameters for Interpretable Deep Classification

Abstract

Deep learning models lack intrinsic interpretability and require manual tuning of information-theoretic hyperparameters. This paper presents HIFN-Transformer (HIFN-T), a framework extending the Variational Information Bottleneck through four jointly learnable per-layer parameters: information retention, entropy budget, magnitude scaling, and global information gates. The retention and budget parameters are co-adapted with the network weights via a stop-gradient decoupling that stabilizes training and substantially reduces the manual hyperparameter burden by eliminating per-layer $\beta $ grid search; the global gates learn per-neuron importance weights, and a Transformer encoder captures complex feature interactions via multi-head self-attention. We prove HIFN generalizes standard VIB as a special case and characterize the role of the entropy budget as an early-training stabilizer that prevents variance explosion before convergence. We evaluate HIFN-T on five benchmarks spanning network intrusion detection (CICIDS-2017, NSL-KDD, GothamDataset2025), image classification (MNIST), and speech emotion recognition (EMO-DB), comparing against XGBoost, TabNet, FT-Transformer, and four additional baselines. HIFN-T delivers consistent improvements across all five. On the official NSL-KDD KDDTrain $+ \,\, \rightarrow $ KDDTest+ benchmark, HIFN-T with a 4-method consensus feature-selection front-end achieves 80.22% accuracy and 0.6898 macro F1, lying within the established 78–87% band of published deep models on this cross-distribution split. On the large-scale GothamDataset2025 IoT benchmark, we evaluate HIFN-T on the full 35,126,611-row release under natural class prevalence as a scalability demonstration on this recently published dataset. MNIST is included as a domain-generalization sanity check rather than a competitive benchmark. Gate-derived feature importance correlates strongly with SHAP analysis across all five datasets, validating intrinsic interpretability. We report all results as mean ± standard deviation over ten independent runs with Friedman and Wilcoxon signed-rank statistical significance tests.