ARMDIL is an ensemble that uses a multimodal large language model (MLLM) agent to dynamically route each image to the most suitable vision backbone, drastically improves adaptability by allowing new information to be integrated via simple prompt modifications, while enhancing interpretability through natural language reasoning traces.
Abstract
Modern image classification models excel when trained on single task-specific datasets but often struggle to generalize across domains and difficulty levels. We propose ARMDIL, an Adaptive Router for Multi-Domain Image Classification with LLMs. ARMDIL is an ensemble that uses a multimodal large language model (MLLM) agent to dynamically route each image to the most suitable vision backbone. Our diverse ensemble employs convolutional neural networks (ResNets), self-supervised representation learners (SSL), and vision language models (VLMs), each trained on a unified label space constructed from multiple image datasets with differing distributions and characteristics. Empirical evaluations illuminate the distinct capabilities and vulnerabilities of each architecture across disparate visual domains. Crucially, we show that ARMDIL effectively navigates these tradeoffs, performing competitively with specialized training-based routers. Furthermore, it drastically improves adaptability by allowing new information to be integrated via simple prompt modifications, while enhancing interpretability through natural language reasoning traces. These advances in cross-dataset image classification pave the way for more reliable general-purpose vision systems such as AI assistants and autonomous robots.
This study systematically compares seven pre-trained feature extractors across three architectural families, convolutional neural networks (CNNs), Vision Transformers (ViTs), and self-supervised models to provide practical guidance on model selection for downstream deep learning tasks.
Rafeek Sibrikhan, M. Mufassirin· Sri Lankan Journal of Techno...· 0 citations
A novel adaptive fusion framework that adaptively combines CNN and Transformer features through learnable gating, attention-based feature integration, and explainable-AI methods is developed, intended to improve both computational efficiency and model interpretability.
Komal Sharma, Monika Sainger· International journal of com...· 0 citations
A unified backbone-expert framework with a common convolutional state-space backbone and two specialized interfaces for automatic modulation recognition, confirming the benefit of expert-interface decoupling over one-size-fits-all architectures.
Vision-language models (VLMs) such as CLIP enable zero-shot classification by comparing image features with text prompts in a shared embedding space. A fundamental property underlying this capability is the global comparability of logits across arbitrary candidate classes. However, VLMs are often adapted to fine-grained domains using techniques such as LoRA. While this improves in-domain accuracy, out-of-domain accuracy degrades. This leads to a highly fragmented model ecosystem, with thousands of specialized models. Multi-Expert-Domain classification seeks to address this problem, by merging LoRAs trained independently on specialized domains. However, due to the independent training, the various domain experts no longer produce globally calibrated logits. As a result, when evaluating over the union of multiple domain-specific class sets, heterogeneous logit scales induce cross-domain interference and artificially high confidence for out-of-domain classes, inducing prediction errors. In this work, we identify domain supervision and cross-domain logit miscalibration as the key issue to scalable multi-domain zero-shot recognition. We propose MED-DSLC, combining domain supervised training and domain-wise logit scaling, to explicitly restore global logit comparability. MED-DSLC is a lightweight solution for MED classification, which is shown to preserve within-domain discrimination while reducing cross-domain logit interference with minimal data. Extensive experiments across diverse fine-grained benchmarks demonstrate that it substantially improves mean accuracy (+15\%), cross-domain robustness, and scalability in the size of MED classification problem. Our results show that restoring output-level calibration is essential under highly data imbalanced settings for achieving a truly zero-shot VLM under multi-domain specialization.
Introduction The automated classification of dermoscopic skin lesions is inherently challenging due to pronounced class imbalance, minimal inter-class variance, visual similarity across lesion types, and the requirement for clinically interpretable predictive outcomes. Methods The present study designed a heterogeneous Vision Transformer ensemble framework for seven-class skin lesion classification using the HAM10000 dataset. The framework integrates three architecturally distinct backbones Swin-Tiny, ViT-Base, and DeiT-Small enhanced with a novel Regional Attention Wrapper (RAW) for spatially selective feature aggregation. The generated outputs are combined via a stacking protocol wherein a trained MLP meta-learner resolves class-aware disagreements among the models. Class imbalance is addressed using class-adaptive augmentation with class-weighted focal loss and MixUp regularisation. Results The proposed framework achieved 98.37% accuracy, weighted F1-score of 98.39%, and mean AUC of 0.999, surpassing all three individual backbones across all metrics. MEL misclassifications were reduced by 78% compared to the weakest baseline, confirmed by McNemar's test (p < 0.0001). Comprehensive ablation studies validate the contributions of the RAW module, each ensemble component, and each augmentation strategy. Three-fold cross-validation yields a mean accuracy of 96.14 ± 0.36% and bootstrap confidence interval analysis confirms the reliability and reproducibility of the reported results. Discussion An exhaustive explainability framework comprising Regional Attention Maps, GradCAM++, SHAP, and t-SNE provides complementary spatial, gradient-based, pixel-level, and embedding-level interpretability, ensuring clinical trust, transparency, and trustworthiness expected from an automated dermoscopy system.
N. Sravani, Srinivas Koppu· Frontiers in Medicine· 0 citations
VisionDES is introduced, a novel DES framework for image classification that uses deep model embeddings to estimate classifier competence and consistently outperforms static ensembles and existing uncertainty-based DES methods, improving robust accuracy under strong attacks and under distribution shifts.
Firuz Juraev, Mohammad Abuhamad, Shaker El-Sappagh et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
