Technical Report No. 290 - Abstract

Stefan Falkner, Aaron Klein, Frank Hutter.
Combining Hyperband and Bayesian Optimization.

Proper hyperparameter optimization is computationally very costly for expensive machine learning methods, such as deep neural networks; the same holds true for neural architecture search. Recently, the bandit-based strategy Hyperband has shown superior performance to vanilla Bayesian optimization methods that are limited to the traditional problem formulation of expensive blackbox optimization. However, while Hyperband has strong anytime performance for finding configurations with acceptable results, it relies on random search and therefore does not find the best configurations quickly. We propose to combine Hyperband with Bayesian optimization by maintaining a probabilistic model that captures the density of good configurations in the input space and samples from this model instead of sampling uniformly at random. We empirically show that our new method combines Hyperband’s strong anytime performance with the strong eventual performance of Bayesian optimization.