Generating classifier committees by stochastically selecting both attributes and training examples

Abstract

Boosting and Bagging, as two representative approaches to learning classifier committees, have demonstrated great success, especially for decision tree learning. They repeatedly build different classifiers using a base learning algorithm, by changing the distribution of the training set. Sasc, as a different type of committee learning method, can also significantly reduce the error rate of decision trees. It generates classifier committees by stochastically modifying the set of attributes but keeping the distribution of the training set unchanged. It has been shown that Bagging and Sasc are, on average, less accurate than Boosting, but the performance of the former is more stable than that of the latter in terms of less frequently obtaining significantly higher error rates than the base learning algorithm. In this paper, we propose a novel committee learning algorithm, called SascBag, that combines Sasc and Bagging. It creates different classifiers by stochastically varying both the attribute set and the distribution of the training set. Experimental results in a representative collection of natural domains show that, for decision tree learning, the new algorithm is, on average, more accurate than Boosting, Bagging, and Sasc. It is more stable than Boosting. In addition, like Bagging and Sasc, SascBag is amenable to parallel and distributed processing while Boosting is not. This gives SascBag another advantage over Boosting for parallel machine learning and datamining.