🍌 BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search

Neural architecture search (NAS) is one of the hottest research areas in machine learning, with hundreds of papers released in the last few years (see this website). In neural architecture search, the goal is to use an algorithm (sometimes even a neural network) to learn the best neural architecture for a given dataset.

The most popular techniques for NAS include reinforcement learning, evolutionary algorithms, Bayesian optimization, and gradient-based methods. Each technique has its strengths and drawbacks. For example, Bayesian optimization (BayesOpt) is theoretically one of the most promising methods, and has seen huge success in hyperparameter optimization for ML, but it is very challenging to run Bayesian optimization for NAS in practice. Bayesian optimization works by modeling the space of neural architectures, and then automatically telling you which neural architecture to try next. See our previous blog post for an introduction to BayesOpt for NAS. However, setting up BayesOpt for NAS requires a huge amount of human effort in creating a hand-crafted distance function and tuning a Gaussian Process.

article: https://arxiv.org/abs/1910.11858

code: https://github.com/naszilla/bananas

medium: https://medium.com/reality-engines/bananas-a-new-method-for-neural-architecture-search-192d21959c0c