Rethinking Data Efficiency in Industrial Dense Prediction: Pretraining Coherence, Not Inductive Bias, Determines ViTs Low-Data Advantage

UvA-DARE (Digital Academic Repository) Elastic ViTs from Pretrained Models without Retraining

SnapViT: single-shot network approximation for pruned Vision Transformers is introduced, a new post-pretraining structured pruning method that enables elastic inference across a continuum of compute budgets, and a self-supervised importance scoring mechanism that maintains strong performance without requiring retraining or labels.

Architectural Inductive Bias in Knowledge Distillation: Disentangling Representational Similarity from Performance Transfer

Deploying deep learning models on resource-constrained edge devices necessitates a trade-off between computational efficiency and robustness. While Knowledge Distillation (KD) is widely used to compress large models into lightweight students, the impact of the Teacher’s architectural inductive bias on the Student’s safety profile remains underexplored. This paper investigates Cross-Architecture Distillation, specifically examining whether the robustness properties of a Vision Transformer (ViT) Teacher can be transferred to a Convolutional Neural Network (CNN) Student (MobileNetV2). Using Centered Kernel Alignment (CKA) and Fourier Spectral Analysis, we demonstrate that robustness transfer is driven by the learning of global shape biases— preserving the teacher's functional geometry—rather than strict feature mimicry. To isolate architectural bias from model capacity confounds and validate scalability to high-resolution data, our experiments are conducted on ImageNet-1k. Results reveal that a MobileNetV2 distilled from a ViT-B/16 Teacher achieves a significantly lower mean corruption error (mCE) compared to one distilled from a massive, capacity-matched ResNet-152 (p < 0.0001, Cohen's d = 21.50), despite comparable clean accuracy. Furthermore, we show that the ViT-distilled student effectively suppresses high-frequency noise, validating its resilience against adversarial perturbations. These findings establish a pareto-optimal strategy for deploying robust, lightweight models, proving that the choice of Teacher architecture is a critical hyperparameter for safety-critical edge applications.

Distilling and Scaling Hierarchical Vision Transformer to 30B Parameters

Same Predictions, Different Reasons: The Effect of Quantization on Model Explanations

A systematic evaluation on how static PTQ affects the interpretability / explainability of five widely used CNN architectures shows that architecture selection is as important as the quantization strategy, and shows that classification accuracy is not a reliable indicator of interpretability stability under reduced precision.

Comparative Performance Analysis of Vision Transformer (ViT) and Convolutional Neural Network (CNN) Architectures for Semarang Batik Motif Classification

Vision Transformers (ViT) capture global image context through self-attention but are data-hungry, typically underperforming Convolutional Neural Networks (CNNs) on the small datasets common in fine-grained tasks such as batik motif recognition. This study investigates whether a ViT, trained via knowledge distillation using the Data-efficient Image Transformer (DeiT), can overcome this limitation and compete with CNNs on a small Semarang Batik dataset. A distilled DeiT-Tiny student learns from a ResNet-50 CNN teacher and is benchmarked against two CNN references: ResNet-50 itself (a substantially larger model) and EfficientNet-B0 (a parameter-matched counterpart). In establishing this comparison, we first uncover a critical dataset integrity issue: the publicly available Semarang Batik Dataset (3,020 images) originates from only 18 unique source photographs, each augmented approximately 167 times prior to publication. This near-duplication causes severe data leakage under conventional random splitting, inflating the test accuracy of all models to a misleading 100% and rendering such evaluation meaningless. We therefore introduce a source-aware splitting strategy that enforces group-level separation between training, test partitions, and evaluate all models across three random seeds for statistical reliability. Under this corrected protocol, the distilled DeiT-Tiny attains the highest mean accuracy (95.18 ± 0.30%) and the lowest variance among the three models, matching both the larger ResNet-50 (94.87%) and the parameter-matched EfficientNet-B0 (94.68%) while using only 5.5M parameters. These results confirm knowledge distillation enables a compact Vision Transformer to compete CNNs on a limited fine-grained dataset, and underscore that verifying sample independence is a prerequisite for trustworthy evaluation on pre-augmented public datasets.

DISTAL: Distillation and Self-Supervised Pretraining for Structure-Agnostic Materials Property Prediction

Materials property prediction remains difficult in low-data settings, where many target properties are supported by only a limited number of labeled samples. Models with the strongest predictive accuracy often depend on crystal structures, which restricts their use in early-stage screening when structural information is limited or unavailable. To address this challenge, we propose DISTAL, a dual-prior framework for structure-agnostic materials property prediction that combines self-supervised compositional pretraining with structure-aware knowledge distillation. DISTAL first learns transferable compositional representations from a large virtual composition space using 145 composition-derived descriptors. It then distills structural knowledge from a pretrained ALIGNN teacher into a composition-conditioned student. This setting allows structural priors to be used during training without requiring structural inputs at inference. By integrating explicit compositional descriptors, pretrained latent features, and distilled structural features within a unified prediction pipeline, DISTAL captures complementary signals that are difficult to recover from any single representation alone. Across 39 benchmark tasks, the best-performing multimodal configuration combines all three signals, and improves over the reference benchmark on 37 tasks. DISTAL achieves the strongest overall performance among all evaluated feature combinations. These results indicate that compositional pretraining and structural distillation provide complementary priors and offer a practical route to robust composition-only prediction in small-data materials informatics. The source code and the pre-trained models are anonymously available at: https://osf.io/eq96d/overview?view_only=451617f42f7849e08750bd1852b48980 and will be released at the official link after acceptance.