Definition

MultiBoosting (Webb 2000) is an approach to multistrategy ensemble learning that combines features of AdaBoost and Bagging. The insight underlying MultiBoosting is that the primary effect of AdaBoost is bias reduction, while the primary effect of bagging is variance reduction. By combining the two techniques, it is possible to obtain both bias and variance reduction, the cumulative effect often being a greater reduction in error than can be obtained with the equivalent amount of computation by either AdaBoost or Bagging alone. Viewed from another perspective, as the size of an ensemble formed by either AdaBoost or Bagging is increased, each successive addition to the ensemble has decreasing effect. Thus, if the benefit of the first few applications of AdaBoost can be combined with the benefit of the first few applications of Bagging, the combined benefit may be greater than simply increasing the number of applications of one or the other.

MultiBoosting. Tablel MultiBoost Algorithm

MultiBoost

input:

  • S0, a sequence of m labeled examples 〈(x1, y1), …, (x m , y m )〉 with labels y j ∈ Y.

  • base learning algorithm BaseLearn.

  • integer T specifying the number of iterations.

  • vector of integers I i specifying the iteration at which each subcommittee i ≥ 1 should terminate.

  1. 1.

    S1 = S0 with instance weights assigned to be 1.

  2. 2.

    set k = 1.

  3. 3.

    For t = 1 to T

  4. 4.

    If I k = t then

  5. 5.

    reweight S t .

  6. 6.

    increment k.

  7. 7.

    C t = Base Learn(S′).

  8. 8.

    \(\epsilon _{t} = \frac{\varSigma _{x_{j}\in S_{t}:\mathrm{C}_{t}(x_{j})\neq y_{j}}weight(x_{j})} {m}\).

  9. 9.

    if ε t > 0. 5 then

  10. 10.

    reweight S t .

  11. 11.

    increment k.

  12. 12.

    go to 7.

  13. 13.

    otherwise if ε t = 0 then

  14. 14.

    set β t to 10−10

  15. 15.

    reweight S t .

  16. 16.

    increment k.

  17. 17.

    otherwise,

  18. 18.

    \(\beta _{t} = \frac{\epsilon _{t}} {(1-\epsilon _{t})}\).

  19. 19.

    St+1 = S t .

  20. 20.

    For each x j ∈ St+1,

  21. 21.

    divide weight (x j ) by 2ε t if C t (x j ) ≠ y j and 2 (1 −ε t ) otherwise.

  22. 22.

    if weight (x j ) < 10−8, set weight (x j ) to 10−8

Output the final classifier: \(C^{{\ast}}(x) = argmax_{y\in Y }\sum _{t;\mathrm{C}_{t}(x)=y}log\frac{1} {\beta _{t}}\).

Algorithm

MultiBoosting operates by dividing the ensemble of classifiers that is to be created into a number of subcommittees. Each of these subcommittees is formed by Wagging (Baner and Kohavi 1999), a variant of Bagging that utilizes weighted instances and, hence, is more readily integrated with AdaBoost. The ensemble is formed by applying AdaBoost to these subcommittees. The resulting algorithm is presented in Table 1. The learned ensemble classifier is C, and the tth member of the ensemble is C t . Each S t is a vector of n weighted training objects whose weights always sum to n. The weights change from turn to turn (the turns indicated by the subscript t). The base training algorithm BaseLearn should more heavily penalize errors on training instances with higher weights. ɛ t is the weighted error of C t on S i . β t is a weight assigned to the tth classifier, Ct. The operation rewieght S t sets the weights of the objects in S t to random values drawn from the continuous Poisson distribution and then standardizes them to sum to n. The code set with a grey background is the code added to AdaBoost in order to create MultiBoost.

Cross-References