TL;DR
Deeper modern networks outperform the older AlexNet by a wide margin on CIFAR-10, and even a relatively compact ResNet can nearly match the accuracy of a much larger VGG16 in far less time.
In this paper, we perform a systematic empirical study of deep learning techniques on the CIFAR-10 image classification benchmark. We study three inter-related aspects of neural network design: (1) the relative impact of nine gradient descent optimisation algorithms on a baseline Multi-Layer Perceptron (MLP); (2) the effectiveness of six regularisation techniques namely; L2 weight decay, Dropout, Data Augmentation, Gaussian Noise Injection, Early Stopping, and Ensemble Learning; on model generalisation; and (3) a benchmark comparison of five landmark Convolutional Neural Network (CNN) architectures alongside Inception V3 trained from scratch. We also investigate autoencoder-based image reconstruction and a side-by-side comparison of linear Principal Component Analysis (PCA) with non-linear autoencoder representations. Nesterov Accelerated Gradient achieves the lowest training loss (1.5814) in short-run experiments. VGGNet achieves the highest test accuracy (75.09%) among CNN architectures trained from scratch for five epochs. Early Stopping yields the best regularisation outcome, and non-linear autoencoders outperform PCA in reconstruction quality. All experiments use TensorFlow 2.x and Keras on CIFAR-10.
A comparative study of different deep learning architectures, including classical CNNs, deep hierarchical models, residual and dense networks, and compound-scaled architectures is presented, showing that deeper networks provide better representation, while residual connections and compound scaling improve training stability and efficiency.
The results demonstrate that transfer learning significantly improves convergence speed, generalization, and computational efficiency, making it a promising approach for AI applications across domains such as healthcare, NLP, and autonomous systems.
It is demonstrated that hyperparameter optimization dynamics depend heavily on dataset complexity, where computational efficiency is the primary differentiator for simpler classification tasks, but optimization architecture selection becomes critical for navigating challenging medical imaging applications.
Modern deep networks often rely on attention modules, which are still at a modest level due to using either one type of channel-wise pattern or an expensive combination of two types of them. In the case of using all of those, the obtained weights can be less discriminative due to the disjointed excitations, while the model complexity would double. To deal with these limitations, an efficient attention is proposed by addressing two novel components of residual attentive information as follows: 1) top- $n$ channel-residual attentive patterns with a unitary excitation perceptron, and 2) multiple spatial-residual attentive features. A simple fusion of these complementary components forms a robust volume of selective residual attentive patterns (named SRAP). Experiments on benchmark datasets for image classification have proved the prominent performance of SRAP versus other attention modules. Particularly, SRAP boosted the performance of lightweight CNN-based networks by up to ~7% on ImageNet-100 without increasing the computational complexity. The implementation code of SRAP is available at https://github.com/nttbdrk25/SRAP.
Findings indicate a trade-off between computational efficiency and predictive performance under the experimental conditions used and show that the fine-tuned VGG16 model consistently outperformed the plain CNN across the principal performance measures.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
