Boosting Algorithms as Gradient Descent

作者：Llew Mason, Jonathan Baxter, Peter L. Bartlett, Marcus R. Frean · 年份：1999 · 被引用次数：707 · 研究领域：Machine Learning and Algorithms、Stochastic Gradient Optimization Techniques、Machine Learning and ELM

Much recent attention, both experimental and theoretical, has been focussed on classification algorithms which produce voted combinations of classifiers. Recent theoretical work has shown that the impressive generalization performance of algorithms like AdaBoost can be attributed to the classifier having large margins on the training data. We present an abstract algorithm for finding linear combinations of functions that minimize arbitrary cost functionals (i.e functionals that do not necessarily depend on the margin). Many existing voting methods can be shown to be special cases of this abstract algorithm. Then, following previous theoretical results bounding the generalization performance of convex combinations of classifiers in terms of general cost functions of the margin, we present a new algorithm (DOOM II) for performing a gradient descent optimization of such cost functions. Experiments on several data sets from the UC Irvine repository demonstrate that DOOM II generally outper...