Simultaneous Multi-Objective Evolutionary Optimization of Heterogeneous Ensembles, Learner-Specific Feature Subsets, and Aggregation Weights

Abstract

This paper introduces an integrated multi-objective evolutionary framework for synthesis of heterogeneous regression ensembles featuring localized, learner-specific feature selection. Rather than enforcing global feature spaces, the proposed paradigm simultaneously optimizes base estimator activation patterns, customized variable subsets tailored to each active learner, and continuous voting weights within a unified mixed-variable optimization process. The evolutionary pipeline minimizes two conflicting axes: predictive error, quantified via Root Mean Squared Error, and structural complexity, modeled as the average cardinality of selected features across active estimators. To prevent data leakage, the framework is validated under a rigorous nested cross-validation architecture using five real-world application benchmarks and comprehensively evaluated against 16 baseline configurations, including standalone learners, static voting ensembles, and isolated wrapper multi-objective feature selection pipelines. The empirical findings demonstrate that our joint evolved-weight variant achieves the dominant global predictive ranking across both non-parametric Wilcoxon and absolute mean trajectories. Concurrently, it maintains exceptional structural parsimony by yielding competitive dimensionality reductions, effectively balancing execution runtimes against Pareto-optimal generalization. Furthermore, a systematic ablation analysis isolates the continuous weighting mechanism as a pivotal driver for discovering significantly more compact ensemble topologies.