Benchmarking Analysis of Evolutionary Neural Architecture Search

Evolutionary computation-based neural architecture search (ENAS) is a popular technique for automating the architecture design of deep neural networks. For any evolutionary computation-based algorithm, the runtime and convergence are the most important aspects concerned by theoretical analysis. However, because of the lacking of benchmarked fitness functions specialized for ENAS, the corresponding theoretical work is rarely available. To address this issue, we propose three different benchmark functions in this paper based on NAS-Bench-101. Specifically, we first propose a correlation-based feature extraction method, to capture the accuracy relationship between neural architectures and their fitness values. Furthermore, we propose a function toolkit, which allows combining different architecture features to specific benchmark functions. In addition, three benchmark functions are derived upon the toolkit by considering the features of neural net topologies, the features of neural operations, and their combinations. Based on these designs, the search space partition and transition probability calculation could be easily established, which in turn greatly promote the runtime and convergence analysis. We perform the experiments of ranking correlation, and the experimental results demonstrate the correctness of the proposed benchmark functions. To the best of our knowledge, this is the first work focusing on ENAS benchmark functions.