Definition
MultiBoosting (Webb 2000) is an approach to multistrategy ensemble learning that combines features of AdaBoost and Bagging. The insight underlying MultiBoosting is that the primary effect of AdaBoost is bias reduction, while the primary effect of bagging is variance reduction. By combining the two techniques, it is possible to obtain both bias and variance reduction, the cumulative effect often being a greater reduction in error than can be obtained with the equivalent amount of computation by either AdaBoost or Bagging alone. Viewed from another perspective, as the size of an ensemble formed by either AdaBoost or Bagging is increased, each successive addition to the ensemble has decreasing effect. Thus, if the benefit of the first few applications of AdaBoost can be combined with the benefit of the first few applications of Bagging, the combined benefit may be greater than simply increasing the number of applications of one or the other.
MultiBoosting. Tablel MultiBoost Algorithm
MultiBoost
input:
-
S0, a sequence of m labeled examples 〈(x1, y1), …, (x m , y m )〉 with labels y j ∈ Y.
-
base learning algorithm BaseLearn.
-
integer T specifying the number of iterations.
-
vector of integers I i specifying the iteration at which each subcommittee i ≥ 1 should terminate.
-
1.
S1 = S0 with instance weights assigned to be 1.
-
2.
set k = 1.
-
3.
For t = 1 to T
-
4.
If I k = t then
-
5.
reweight S t .
-
6.
increment k.
-
7.
C t = Base Learn(S′).
-
8.
\(\epsilon _{t} = \frac{\varSigma _{x_{j}\in S_{t}:\mathrm{C}_{t}(x_{j})\neq y_{j}}weight(x_{j})} {m}\).
-
9.
if ε t > 0. 5 then
-
10.
reweight S t .
-
11.
increment k.
-
12.
go to 7.
-
13.
otherwise if ε t = 0 then
-
14.
set β t to 10−10
-
15.
reweight S t .
-
16.
increment k.
-
17.
otherwise,
-
18.
\(\beta _{t} = \frac{\epsilon _{t}} {(1-\epsilon _{t})}\).
-
19.
St+1 = S t .
-
20.
For each x j ∈ St+1,
-
21.
divide weight (x j ) by 2ε t if C t (x j ) ≠y j and 2 (1 −ε t ) otherwise.
-
22.
if weight (x j ) < 10−8, set weight (x j ) to 10−8
Output the final classifier: \(C^{{\ast}}(x) = argmax_{y\in Y }\sum _{t;\mathrm{C}_{t}(x)=y}log\frac{1} {\beta _{t}}\).
Algorithm
MultiBoosting operates by dividing the ensemble of classifiers that is to be created into a number of subcommittees. Each of these subcommittees is formed by Wagging (Baner and Kohavi 1999), a variant of Bagging that utilizes weighted instances and, hence, is more readily integrated with AdaBoost. The ensemble is formed by applying AdaBoost to these subcommittees. The resulting algorithm is presented in Table 1. The learned ensemble classifier is C, and the tth member of the ensemble is C t . Each S t is a vector of n weighted training objects whose weights always sum to n. The weights change from turn to turn (the turns indicated by the subscript t). The base training algorithm BaseLearn should more heavily penalize errors on training instances with higher weights. ɛ t is the weighted error of C t on S i . β t is a weight assigned to the tth classifier, Ct. The operation rewieght S t sets the weights of the objects in S t to random values drawn from the continuous Poisson distribution and then standardizes them to sum to n. The code set with a grey background is the code added to AdaBoost in order to create MultiBoost.
Cross-References
Recommended Reading
Bauer E, Kohavi R (1999) An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Mach Learn 36(1):105–139
Webb GI (2000) MultiBoosting: a technique for combining boosting and wagging. Mach Learn 40(2):159–196
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer Science+Business Media New York
About this entry
Cite this entry
Webb, G.I. (2017). MultiBoosting. In: Sammut, C., Webb, G.I. (eds) Encyclopedia of Machine Learning and Data Mining. Springer, Boston, MA. https://doi.org/10.1007/978-1-4899-7687-1_567
Download citation
DOI: https://doi.org/10.1007/978-1-4899-7687-1_567
Published:
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4899-7685-7
Online ISBN: 978-1-4899-7687-1
eBook Packages: Computer ScienceReference Module Computer Science and Engineering