Hyperparameter optimization

In machine learning, hyperparameter optimization^[1] or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. A hyperparameter is a parameter whose value is used to control the learning process, which must be configured before the process starts.^[2][3]

Hyperparameter optimization determines the set of hyperparameters that yields an optimal model which minimizes a predefined loss function on a given data set.^[4] The objective function takes a set of hyperparameters and returns the associated loss.^[4] Cross-validation is often used to estimate this generalization performance, and therefore choose the set of values for hyperparameters that maximize it.^[5]