A dual-objective surrogate-guided ensemble search is introduced: candidate architectures are represented as directed acyclic graphs, and two surrogate models are trained independently to estimate predictive accuracy and diversity potential and guide an NES framework that efficiently identifies architectures that are both individually strong and collectively diverse.
Abstract
Ensembles are a standard way to improve the performance and robustness of deep neural networks, but their effectiveness crucially depends on both the quality and the diversity of individual models. Most neural architecture search (NAS) methods are computationally expensive. Extending them to neural ensemble search (NES), which requires joint optimization of individual architectures and their ensemble composition, leads to an exponential growth of the search space and makes the problem computationally intractable. To address this, we introduce a dual-objective surrogate-guided ensemble search: candidate architectures are represented as directed acyclic graphs, and two surrogate models are trained independently to estimate predictive accuracy and diversity potential. Their combined estimates guide an NES framework that efficiently identifies architectures that are both individually strong and collectively diverse. Our final ensemble achieves competitive or superior performance compared to standard baselines such as Deep Ensembles and Random Search on FashionMNIST, CIFAR-10, and CIFAR-100.
Deep neural networks can achieve very high accuracy, but their architectures are most often designed by hand and tuned for a single goal, such as accuracy. In practice, we often care about several goals at once, for example, accuracy, model size, and computational cost. Multi-objective neural architecture search (MONAS) can find a set of architectures that balance these goals, but the resulting models are usually very different from each other, which makes them hard to maintain and deploy as a family. In this work, we study how to make the diverse trade-off architectures to possess a regularity, so they can be better understood, maintained, and deployed with confidence. To this effort, we first run a standard multi-objective NAS to obtain a set of Pareto neural architectures, then analyze the common structural patterns that appear frequently among them. We then fix these common patterns and run a second MONAS in the reduced search space to obtain a set of regularized neural architectures. Across NAS-Bench-101, NAS-Bench-201, and an AG News text classification task, we show that this regularity-driven search can produce families of architectures that remain competitive in performance while being structurally simpler and having a regular pattern.
Ritam Guha, Shashank Raj, Kalyanmoy Deb· Proceedings of the Genetic a...· 0 citations
We introduce an efficient Bayesian deep ensemble method for predictive regression designed to enhance interpretability while maintaining competitive predictive performance and computational efficiency. Our method combines the statistical rigor of Bayesian inference with the scalability of deep ensembles, providing calibrated uncertainty estimates that enable its use not only for standalone prediction but also as a component within broader learning systems. To achieve these goals, our work relies on three key design components: (i) low-dimensional ensemble representation: predictions are expressed as a combination of a small number of trained neural predictors, enabling scalable inference whose cost depends on ensemble size rather than dataset size; (ii) closed-form Bayesian aggregation: ensemble predictions are combined using Bayesian linear regression, yielding interpretable posterior weights and calibrated uncertainty without approximate inference; and (iii) Independent ensemble training: multiple neural networks are trained separately, producing diverse predictive representations that improve robustness and uncertainty calibration. Empirical results on standard regression benchmarks demonstrate that the proposed approach achieves competitive predictive performance while maintaining reliable uncertainty estimates across settings.
Sina Aghaee Dabaghan Fard, M. Maros, Jaesung Lee· 0 citations
A normalisation-based implicit ensemble that treats each member as a task in a multi-task architecture and modulates the shared backbone through sigmoid-bounded scalers is introduced, which matches or outperforms deep ensembles at a fraction of their parameter cost, scales with ensemble size where partitioning methods collapse, and maintains calibration under distribution shift.
Mihai Suteu, Ovidiu Serban· arXiv.org· 0 citations
Results show that surrogate-guided search can substantially reduce the evaluation cost of gradient-free post-training while producing stronger deployable single experts.
Nigel Bastian Cendra, Abdelhamid Ezzerg, Fernando Julio Cendra et al.· 0 citations
Neural Architecture Search (NAS) aims to identify high-performance networks within a defined search space. Training-free metrics have been proposed to estimate network performance without actual training, reducing NAS deployment costs. However, individual training-free metrics often capture only partial architectural features, and their estimation capabilities are different in various tasks. Combining multiple training-free metrics has been explored to enhance scalability across tasks. Yet, these methods typically optimize global metric combinations over the entire search space, overlooking the varying sensitivities of different architectures to specific metrics, which may limit the final architectures’ performance. To address these challenges, we propose the Per-Architecture Training-Free Metric Optimization NAS (PO-NAS) algorithm. This algorithm: (a) Integrates multiple training-free metrics as auxiliary scores, dynamically optimizing their combinations using limited real-time training data, without relying on benchmarks; (b) Individually optimizes metric combinations for each architecture; (c) Integrates an evolutionary algorithm that leverages efficient predictions from the surrogate model, enhancing search efficiency in large search spaces. Notably, PO-NAS combines the efficiency of training-free search with the robust performance of training-based evaluations. Extensive experiments demonstrate the effectiveness of our approach. Our code has been made publicly available at https://github.com/LMZ-Zhuo/PO-NAS .
Mingzhuo Lin, Jianpin Luo· Neural Information Processin...· 2 citations
It is demonstrated that hyperparameter optimization dynamics depend heavily on dataset complexity, where computational efficiency is the primary differentiator for simpler classification tasks, but optimization architecture selection becomes critical for navigating challenging medical imaging applications.
Sarab Almuhaideb, Ahmad Raza Khan· Applied Sciences· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
