Dynamic Context-Aware Neural Architecture Search (DCA-NAS)

Abstract

This paper introduces Dynamic Context-Aware Neural Architecture Search (DCA-NAS), a novel approach to neural architecture search that addresses the limitations of traditional NAS methods by incorporating dynamic contextual information during the search process. The core claim of DCA-NAS is that real-time analysis of intermediate representations generated during training—such as activation values and gradient information—can dynamically adjust the exploration strategy of the NAS algorithm, thereby accelerating the discovery of optimal architectures. DCA-NAS combines generative NAS techniques (e.g., evolutionary algorithms, reinforcement learning) with search-based NAS (e.g., differentiable architecture search) through an integrated "context module." This module learns to identify the contextual factors most influential on architecture selection, considering not only the statistical properties of the searched architecture but also the statistical properties of intermediate representations, training loss gradients, and learning rates. Based on this contextual understanding, the module dynamically adjusts the search strategy of the generative component or the search space of the search-based component. Furthermore, DCA-NAS divides the search process into multiple "sub-search" stages, each optimized for a specific context. Experimental results demonstrate DCA-NAS's ability to achieve superior architecture quality and search efficiency compared to static NAS methods, particularly in scenarios with diverse tasks and datasets.