2023· International Journal of Applied Data Science & Modern Computing· 0 citations
TL;DR
Experimental findings show that pretrained models outperform those trained from scratch in terms of accuracy, convergence speed, and robustness, and a unified framework is proposed to integrate both processes in a deep learning pipeline.
Abstract
Deep learning has achieved state-of-the-art performance across domains such as computer vision, NLP, and healthcare, but it often requires large datasets and high computational resources. Pretraining and fine-tuning have emerged as effective strategies to improve performance, reduce training cost, and enhance generalization. Pretraining learns transferable representations from large-scale data, while fine-tuning adapts models to specific tasks with limited labeled data. This paper provides a comprehensive study of various pretraining methods (supervised, unsupervised, self-supervised) and fine-tuning techniques, including full-model and parameter-efficient approaches. A unified framework is proposed to integrate both processes in a deep learning pipeline. Experimental findings show that pretrained models outperform those trained from scratch in terms of accuracy, convergence speed, and robustness. The study also highlights challenges, trade-offs, and emerging trends such as foundation models and multimodal learning, emphasizing the importance of these techniques in advancing deep learning systems.
Transfer learning has become an effective paradigm in machine learning and deep learning, especially in the case when labeled data are limited. Small scale data are a big challenge to the conventional deep learning models since they overfit, are not able to generalize, and do not learn features well. Transfer learning helps to address these problems by using the information of large domains in the source task to facilitate activities in target domains with scarce data. In this paper, the researcher will take the task of thoroughly examining transfer learning methods with small-scale datasets in mind. Our view of the base concepts, architecture, and strategies of domain adaptation techniques, which make the reuse of pretrained models effective are analyzed. The literature review carried out in the paper examines available literature indicating supervised, unsupervised, and semi-supervised transfer learning techniques. In addition, we suggest a systematic approach to the execution of transfer learning pipelines such as feature extraction, fine-tuning schemes, regularization schemes and metrics of evaluation. The practical experience and performance evaluation shows that, transfer learning leads to a high convergence speed, accurate classification, and robustness, in comparison to scratch training. There are negative transfer, domain-shift, and model-selection challenges which are discussed too. Lastly, the paper discusses future research directions in the area of self-supervised learning, few-shot learning, and adaptive transfer mechanisms in the regimes of small data. The results confirm that transfer learning is a core solution to the real-world problems that may be limited by available labeled data.
Sivesh Prasana, Karthick eyan· International Journal of Mac...· 0 citations
The rapid growth of artificial intelligence and deep learning has revolutionized diverse domains, including computer vision where transfer learning through pre-trained models has become a fundamental technique for enhancing performance of models and reducing computational costs. Although numerous pre-trained deep learning models have been widely adopted, their effectiveness varies significantly across application domains and datasets. The lack of comprehensive comparative studies makes it challenging for researchers to identify the most suitable models for achieving optimal performance in specific tasks. This study systematically compares seven pre-trained feature extractors across three architectural families, convolutional neural networks (CNNs), Vision Transformers (ViTs), and self-supervised models to provide practical guidance on model selection for downstream deep learning tasks. These models were evaluated on five benchmark datasets. Features were extracted with frozen weights and evaluated using linear probing, k-nearest neighbor retrieval, and 5-shot classification. ConvNeXt-B achieved the highest mean linear probe accuracy (90.40%), while DINOv2-S produced the best feature geometry for retrieval tasks (87.64%). CLIP-ViT-B/32 demonstrated the strongest cross-domain transfer, leading on texture recognition and satellite imagery few-shot classification. Older CNN architectures lagged significantly behind modern models by approximately 7%. Overall, this study simultaneously evaluates multiple feature extractors across multiple visual domains and evaluation protocols.
Rafeek Sibrikhan, M. Mufassirin· Sri Lankan Journal of Techno...· 0 citations
Transfer learning has become a key technique for improving the accuracy of neural networks in low-resource, low-data environments. The quantitative comparative analysis of the pre-trained models includes ResNet50, VGG16, BERT, GPT, and the baseline CNN and LSTM models. They are compared across three different application areas: computer vision, natural language processing (NLP), and medical imaging. The five benchmark datasets used were ImageNet, CIFAR-10, SST-2, IMDB, and Chest X-Ray. All experiments used the same preprocessing pipeline and evaluation metrics (accuracy, F1 score, precision, recall, and ROC-AUC). Results showed that models trained on the pre-trained data achieved consistently greater accuracy than the baselines in all domains (9-20%) and F1-score (0.09-0.16) gains. ResNet50 achieved 92% accuracy on CIFAR-10, compared to 72% for the CNN baseline, whereas BERT hit 92% on SST-2, with 80% accuracy for LSTM. VGG16 improved the accuracy of Chest X-Ray classification from 78% to 87% and reduced training time by up to 60%. There were a few instances of minor overfitting and domain mismatch, emphasizing the need for adaptive fine-tuning strategies. The results demonstrate that transfer learning significantly improves convergence speed, generalization, and computational efficiency, making it a promising approach for AI applications across domains such as healthcare, NLP, and autonomous systems.
Abdul Sttar Ismail Wdaa, Iraq Ali Hussein, A. Ahmed· Future Technology· 0 citations
Self-supervised learning (SSL) is transforming artificial intelligence by enabling models to learn from large amounts of unlabeled data. Instead of relying on manual annotations, SSL leverages inherent data patterns to generate pseudo-labels, making it highly scalable and efficient for modern AI systems. Techniques such as contrastive learning, masked modeling, generative pretraining, and clustering have shown strong performance across vision, language, and speech tasks. This study examines key SSL methods, architectures, and training strategies, while addressing challenges like computational cost, feature collapse, and data bias. It proposes a unified framework that combines contrastive and generative approaches for improved efficiency and representation learning. Experimental results demonstrate that SSL outperforms traditional supervised learning in accuracy, scalability, and transferability, while also reducing data labeling costs. Future directions include integrating multimodal, reinforcement, and continual learning to further enhance SSL systems.
Shalini Gupta· International Journal of Inn...· 0 citations
Self-supervision is a powerful technique for learning visual representations from unlabeled data. Existing techniques primarily adopt a two-stage approach for self-supervised learning (SSL): a pretraining stage on unlabeled data followed by a finetuning stage on labeled data. While this pipeline has demonstrated extreme effectiveness, the interaction between self-supervised and supervised learning objectives remains insufficiently understood. In this work, we systematically investigate whether jointly optimizing the self-supervised and supervised objectives during training provides a better alternative. We compare two training paradigms: (1) the aforementioned pretraining followed by finetuning (PFT) and (2) joint training (JT), where self-supervised and supervised losses are optimized simultaneously in the same network. Across eight representative SSL methods and diverse computer vision tasks on natural, medical, crisis response, and remote sensing data, we evaluate performance under varying percentages of labeled data. Our results reveal that the relative effectiveness of PFT and JT depends strongly on the task at hand, the availability of labeled data, and the complexity of the domain. We find that JT consistently improves data and training efficiency while being robust in low-label settings, while PFT is more reliable in more specialized domains. We further analyze representation quality, robustness, and cross-domain generalization, providing new insights into how self-supervised and supervised objectives interact during optimization. We establish a comprehensive empirical benchmark for hybrid SSL-based semi-supervised learning and offer practical guidance for selecting appropriate training strategies across diverse vision applications.
Nusrat Munia, Tyler Ward, Nishat Nayla et al.· 0 citations
Passive acoustic monitoring (PAM) generates large datasets that are costly to analyze manually, creating the need for accurate automated detection methods. While deep learning (DL) has shown promise for PAM‐related tasks, the transferability of trained models to unseen datasets remains a critical challenge. To address this, we evaluated four methodological approaches for maximizing the transferability of a DL sperm whale (
Physeter macrocephalus
) click detector: (1)
cross‐environment evaluation
(training on five datasets, testing on an unseen sixth), (2)
limited target‐domain data training
(model retraining using only a small, site‐specific dataset), (3)
pretraining with random fine‐tuning
, and (4)
pretraining with active fine‐tuning
(selecting fine‐tuning samples based on model uncertainty). While trained models saw a drop in performance when tested on a new dataset, those pretrained on large diverse datasets outperformed limited target‐specific training even when deployment conditions were not represented during training, confirming training diversity improves transferability. Fine‐tuning with 500 target recordings effectively mitigated performance drops when deploying a trained model to new datasets. Active fine‐tuning consistently outperformed all other approaches, although the improvement over random fine‐tuning was marginal. For new studies, we recommend using models pretrained on diverse datasets and fine‐tuning them with minimal target data.
Laia Garrobé Fonollosa, D. Gillespie, L. Stanković et al.· Marine mammal science· 0 citations