Curriculum learning applied to the combined algorithm selection and hyperparameter optimization problem

Abstract

AutoML has the goal to find the best Machine Learning (ML) pipeline in a complex and high dimensional search space by evaluating multiple algorithm configurations. Training multiple ML algorithms is time costly, and as AutoML tools usually obey a time constraint, the exploration of the search space may find sub-optimal results. In this work, we explore the application of curriculum learning techniques to overcome this limitation. Curriculum learning and anti-curriculum learning are strategies for ordering examples during model training based on their difficulty. These have shown to improve model performance and accelerate the training process on previous empirical investigations using optimization-based models. We apply and compare curriculum strategies on two optimizers of an AutoML system to accelerate the search space exploration and find good performing machine learning pipelines with efficiency. The results indicate that AutoML can benefit from a curriculum strategy. In most of the evaluated scenarios, the curriculum strategies led the AutoML algorithm to better classification results.