Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks

Research on Neural Network Training Mechanism Integrating Convex Optimization and Backpropagation

This paper deeply integrates convex optimization theory with the backpropagation algorithm and constructs a novel stable and efficient training mechanism for neural networks that achieves favorable adaptability to both shallow fully connected networks and deep convolutional networks.

From Perceptrons to Convolutional Neural Networks: A Practical Tutorial on Spatial Deep Learning

This tutorial takes the reader on a historical and technical journey from the simple Perceptron (1958) to modern Convolutional Neural Networks (CNNs) that dominate spatial data processing (images and video). We start with the Perceptron’s linear classifier, then expose its inability to learn non-linear patterns (e.g., XOR), which motivates the Multi-Layer Perceptron (MLP) and the backpropagation algorithm. Next, we discuss the limitations of MLP when faced with structured data like images—parameter explosion, loss of spatial information, and lack of translation invariance—and use these limitations as a natural springboard to the core ideas of CNNs: local connectivity, weight sharing, and hierarchical feature learning. Throughout, we provide intuitive explanations, mathematical formulations, and step-by-step numerical examples (e.g., a complete forward and backward pass for a small network, and a manual 2D convolution). Clear graphical representations and examples help readers understand each concept. The tutorial concludes with a detailed walkthrough of influential CNN architectures (LeNet-5, AlexNet, VGG, GoogLeNet, ResNet, DenseNet, and EfficientNet) and also discusses more recent attention-based models (e.g., Vision Transformers and ConvNeXt), explaining why each was necessary and how it advanced the field. Aimed at students and practitioners with a basic knowledge of calculus and linear algebra, this tutorial connects foundational ideas to state-of-the-art deep learning, focusing on spatial data. It is designed for readers who want to understand why each architectural choice was made, not just what the final model looks like.

Backpropagation-Free Trunk Training via the Split Forward Gradients

Backpropagation makes training deep networks memory intensive because it must store intermediate activations. Forward-mode methods avoid this cost, but their gradient estimates become increasingly noisy as the number of trained parameters grows. We introduce Split Forward Gradient (Split-FG), which splits a network at an intermediate representation: it computes the output head gradient exactly and estimates only the trunk gradient with a Jacobian--vector product. This reduces estimator variance and requires no backward pass through the trunk, while retaining an Adam-style convergence guarantee. Our experiments reveal an important practical failure mode. On WikiText-103, naive forward-gradient training of the trunk performs worse than leaving a randomly initialized trunk frozen, likely because Adam updates every noisy, under-determined trunk coordinate too aggressively. Simply using a much smaller learning rate for the trunk reverses this result: a $16$M-parameter GPT-2-style model reaches validation perplexity $387$, compared with $668$ for the frozen-trunk control and $2{,}885$ for a matched pure forward-gradient baseline (backpropagation reaches $150$). Split-FG also produces the strongest backprop-free results on our tabular benchmarks and reaches $60.5\%$ on CIFAR-10 and $35.2\%$ on CIFAR-100 with a heavy-head design. It reduces peak memory by up to $35\%$ relative to matched backpropagation, although the performance gap widens as the forward-mode trunk grows.

Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks

A Noise-modulated Neural Network (NNN) learns and infers only in the presence of noise, treating noise as a computational resource rather than a disturbance. The noise lets it learn efficiently by backpropagation while transmitting spike-like signals, but backpropagation needs a reverse path through transposed weights, the weight transport problem, which undermines biological and neuromorphic plausibility. Forward-only alternatives typically substitute a different objective or fixed random feedback, sacrificing stability and accuracy. We show that backpropagation itself can be reconstructed in the NNN from forward-pass statistics alone: a weight mirror estimates each weight matrix from the covariance between a previous-layer unit's output and the next-layer unit's input, and combining it with local differential estimation inside the units propagates the output error recursively along the computational graph, with no transposed-weight readout and no backward data path. The resulting gradient is empirically near-unbiased, and with local per-weight Adam updates it matches the final accuracy of backpropagation on simple regression tasks. With uniformly distributed noise, the local operations reduce to polynomials and comparators, making the whole system, learning rule included, well suited to digital circuits. Thus, in the NNN, noise is a resource not only for inference but also for reconstructing backpropagation.

Efficient Hyperbolic Perceptron for Image Classification

Adaptive Gating Dynamics for Layer-Wise Sparsity Learning and Neural Network Pruning

Deep neural networks often contain substantial parameter redundancy, resulting in unnecessary computational cost and energy consumption. This work presents a dynamically adaptive gating mechanism for learning layer-wise sparsity through differentiable masking. In the proposed framework, each network parameter is associated with a self-learning gate that controls its contribution during training. The gating function follows a progressive soft-to-hard transition in which the slope is gradually annealed, enabling the model to move smoothly from continuous parameter weighting to near-binary pruning decisions while simultaneously learning an adaptive threshold parameter. The framework is evaluated across multilayer perceptrons (MLPs), deep neural networks (DNNs), Tabular Transformer models, and benchmarked against $\mathrm{L}_{0}$ regularization and Variational Dropout on convolutional neural networks. Experimental results demonstrate substantial model compression without degrading predictive performance, and in some cases improving it. On MNIST, the gated MLP retains only 14% of weights (86% pruning) while maintaining 98% accuracy. The gated DNN and Tab Transformer similarly outperform their dense counterparts while retaining only 46% and 46.22% of parameters, respectively. Benchmarking further shows that the proposed adaptive gating achieves competitive or superior accuracy-sparsity trade-offs compared with $\mathbf{L}_{0}$ regularization and Variational Dropout. These results demonstrate that the proposed dynamically adaptive gating framework provides an efficient and interpretable pathway for sparsity learning, enabling high-performing lightweight neural network deployment.