LG AI Research: NeurIPS 2021 Research Summary - Part 3: Neural Architecture Search Review

Neural Architecture Search (NAS) is a field of AutoML that searches for the optimal neural network structure for a specific goal. Previously, humans designed these structures directly, but there was a limitation in that finding an appropriate structure whenever the data or objective changed required significant cost and time.

NAS goes through the following process to find the optimal structure within a Search Space.

• Search Space Definition: Setting selectable options such as activation functions, number of layers, and number of nodes
• Search Strategy: Selecting structures using Reinforcement Learning or Evolutionary Methods
• Performance Estimation: Measuring the performance of the selected structure and feeding it back into the search

The core challenge in current NAS research is reducing the massive cost involved in performance estimation. The following techniques have been proposed to address this.

• Gradient-based NAS Stabilization: Weights warm-up, regularization, and Loss landscape smoothing
• NAS Speed Improvement: Accelerating optimization through Performance predictors, Proxy tasks, and sequential search
• Final Performance Improvement: Deriving optimal structures during the search process, utilizing hyperparameters and data augmentation

In particular, research based on Performance predictors, which predict the ranking of structures by training only a portion of the full space or without training at all instead of training the entire space, is drawing attention.