Aug 2026· Applied Sciences· 0 citations· 38 references
TL;DR
It is demonstrated that hyperparameter optimization dynamics depend heavily on dataset complexity, where computational efficiency is the primary differentiator for simpler classification tasks, but optimization architecture selection becomes critical for navigating challenging medical imaging applications.
Abstract
Hyperparameter optimization is a critical determinant of the performance and generalizability of convolutional neural networks (CNNs), yet selecting optimal configurations remains a complex challenge due to the large, non-linear search space. While numerous optimization techniques have been explored in the literature, most comparative studies have relied on benchmark datasets, limiting insights into their practical effectiveness under real-world constraints. This study evaluates four widely used hyperparameter optimization methods (i.e., Grid Search, Optuna, Hyperopt, and DEAP) across five CNN architectures (i.e., AlexNet, GoogLeNet, LeNet, MobileNet, and ResNet18) under two distinct data regimes: the Modified National Institute of Standards and Technology (MNIST) benchmark and a custom SkinCancer image dataset. On MNIST, all configurations achieved strong classification performance, with test accuracies ranging from 95.45% to 97.61%, led by ResNet18 optimized via Grid Search (97.61%). In this simpler setting, DEAP provided the shortest optimization runtimes, while Optuna and Hyperopt offered an effective balance between performance and computational cost. In contrast, the SkinCancer dataset exhibited substantially greater variability in performance, highlighting the stronger influence of both model architecture and optimization strategy. ResNet18 achieved the highest overall test accuracy (81.50%) when optimized with Hyperopt. Although no single optimization method consistently outperformed all others, adaptive approaches frequently achieved competitive or superior predictive performance while requiring considerably less computation than exhaustive search. These findings demonstrate that hyperparameter optimization dynamics depend heavily on dataset complexity, where computational efficiency is the primary differentiator for simpler classification tasks, but optimization architecture selection becomes critical for navigating challenging medical imaging applications.
Hyperparameter optimization (HPO) can materially affect the performance of deep learning (DL) image classifiers, but there is little empirical guidance on how to derive the validation signal that drives it, especially for the small sample sizes common in fields such as medical imaging. We compared three HPO protocols in terms of {\em absolute performance-estimation error} (AEE; the absolute difference between the winning configuration's validation AUROC and its test AUROC): fixed holdout (F), reshuffled holdout (R), and 5-fold cross-validation (C). The search space, sampler, training procedure, architecture, and test set were held identical across protocols. We evaluated the protocols on three public datasets spanning two regimes: binary medical imaging (RSNA pneumonia radiographs and binarized HAM10000 skin lesions) and 200-class natural imaging (Tiny ImageNet), across a range of development set sizes $n$ and two backbones (ResNet-18 on all datasets, Vision Transformer (ViT-S/16) on RSNA). On the medical datasets, every point estimate favored cross-validation over both holdout protocols, with reductions in AEE largest at small sample sizes and diminishing as $n$ increased. This pattern remained robust under conservative family-wise adjustment. On Tiny ImageNet, AEE was negligible under all three protocols. Test AUROC was generally similar among protocols. Fixed holdout had lower mean AEE than reshuffled holdout in 11 of 12 medical conditions, although this secondary finding was less uniformly supported. For small-sample medical image classification, we recommend cross-validation-based HPO when computational resources permit because it trades additional computation for a more reliable development-time estimate of subsequent test performance.
In this paper, we perform a systematic empirical study of deep learning techniques on the CIFAR-10 image classification benchmark. We study three inter-related aspects of neural network design: (1) the relative impact of nine gradient descent optimisation algorithms on a baseline Multi-Layer Perceptron (MLP); (2) the effectiveness of six regularisation techniques namely; L2 weight decay, Dropout, Data Augmentation, Gaussian Noise Injection, Early Stopping, and Ensemble Learning; on model generalisation; and (3) a benchmark comparison of five landmark Convolutional Neural Network (CNN) architectures alongside Inception V3 trained from scratch. We also investigate autoencoder-based image reconstruction and a side-by-side comparison of linear Principal Component Analysis (PCA) with non-linear autoencoder representations. Nesterov Accelerated Gradient achieves the lowest training loss (1.5814) in short-run experiments. VGGNet achieves the highest test accuracy (75.09%) among CNN architectures trained from scratch for five epochs. Early Stopping yields the best regularisation outcome, and non-linear autoencoders outperform PCA in reconstruction quality. All experiments use TensorFlow 2.x and Keras on CIFAR-10.
Akber Hussain, M. Sajid, Abdul Raheem et al.· 2026 4th International Confe...· 0 citations
Deeper modern networks outperform the older AlexNet by a wide margin on CIFAR-10, and even a relatively compact ResNet can nearly match the accuracy of a much larger VGG16 in far less time.
Neural networks have become an essential tool for solving complex tasks in domains such as image recognition, natural language processing, and autonomous systems. However, the performance of neural networks heavily depends on the choice of hyperparameters such as learning rate, batch size, activation functions, and the number of hidden layers. Hyperparameter optimization, therefore, plays a critical role in enhancing model accuracy, convergence speed, and generalization capabilities. Traditional tuning methods like manual selection and grid search are often computationally expensive and suboptimal for large-scale networks. In this study, we explore advanced hyperparameter tuning strategies including Random Search, Bayesian Optimization, Hyperband, and Genetic Algorithms, evaluating their efficiency and effectiveness on various benchmark datasets. The research presents a comprehensive methodology integrating automated hyperparameter selection with neural network training, highlighting the trade-offs between computational cost and model performance. Experimental results demonstrate that optimized hyperparameters significantly improve the accuracy and stability of neural networks, reducing overfitting and training time. This paper also proposes a systematic framework for hyperparameter optimization that can guide practitioners and researchers in selecting optimal configurations tailored to their specific problem domains. By comparing the performance across different tuning strategies, we offer practical insights into the scalability and adaptability of neural network optimization techniques. The findings underscore the importance of leveraging intelligent hyperparameter optimization methods to advance deep learning applications and achieve superior performance in real-world scenarios.
N. Rahman· International Journal of App...· 0 citations
The results demonstrate that transfer learning significantly improves convergence speed, generalization, and computational efficiency, making it a promising approach for AI applications across domains such as healthcare, NLP, and autonomous systems.
Abdul Sttar Ismail Wdaa, Iraq Ali Hussein, A. Ahmed· Future Technology· 0 citations
The results show that larger models and larger pretraining datasets do not automatically lead to better downstream performance, and transfer effectiveness in medical imaging is driven primarily by architectural inductive biases, pretraining strategy, and domain relevance.
Dina A. Elkholy, Mohamed S. Shehata, John W. Braun· Journal of imaging informati...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
