Abstract
From the beginning of machine learning, rule induction has been regarded as one of the most important issues in this research area. One of the first rule induction algorithms was AQ introduced by Michalski in early 80’s. AQ, as well as several other well-known algorithms, such as CN2 and Ripper, are all based on sequential covering. With the advancement of machine learning, some new techniques based on statistical learning were introduced. One of them, called boosting, or forward stagewise additive modeling, is a general induction procedure which appeared to be particularly efficient in binary classification and regression tasks. When boosting is applied to induction of decision rules, it can be treated as generalization of sequential covering, because it approximates the solution of the prediction task by sequentially adding new rules to the ensemble without adjusting those that have already entered the ensemble. Each rule is fitted by concentrating on examples which were the hardest to classify correctly by the rules already present in the ensemble. In this paper, we present a general scheme for learning an ensemble of decision rules in a boosting framework, using different loss functions and minimization techniques. This scheme, called ENDER, is covered by such algorithms as SLIPPER, LRI and MLRules. A computational experiment compares these algorithms on benchmark data.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Michalski, R.S.: A Theory and Methodology of Inductive Learning. In: Michalski, R.S., Carbonell, J.G., Mitchell, T.M. (eds.) Machine Learning: An Artificial Intelligence Approach, pp. 83–129. Tioga Publishing, Palo Alto (1983)
Clark, P., Niblett, T.: The CN2 induction algorithm. Machine Learning 3, 261–283 (1989)
Cohen, W.W.: Fast Effective Rule Induction. In: Proc. of International Conference of Machine Learning, pp. 115–123 (1995)
Fürnkranz, J.: Separate-and-Conquer Rule Learning. Artificial Intelligence Review 13(1), 3–54 (1996)
Jovanoski, V., Lavrac, N.: Classification Rule Learning with APRIORI-C. In: Proc. of the 10th Portuguese Conference on Progress in Artificial Intelligence, Knowledge Extraction, Multi-agent Systems, Logic Programming and Constraint Solving, London, UK, pp. 44–51. Springer, Heidelberg (2001)
Stefanowski, J., Vanderpooten, D.: Induction of Decision Rules in Classification and Discovery-oriented Perspectives. International Journal on Intelligent Systems 16(1), 13–27 (2001)
Bazan, J.G.: Discovery of Decision Rules by Matching New Objects Against Data Tables. In: Polkowski, L., Skowron, A. (eds.) RSCTC 1998. LNCS (LNAI), vol. 1424, pp. 521–528. Springer, Heidelberg (1998)
Góra, G., Wojna, A.: RIONA: A New Classification System Combining Rule Induction and Instance-based Learning. Fundamenta Informaticae 54, 369–390 (2002)
Domingos, P.: Unifying Instance-based and Rule-based Induction. Machine Learning 24, 141–168 (1996)
Góra, G., Wojna, A.: Local Attribute Value Grouping for Lazy Rule Induction. In: Alpigini, J.J., Peters, J.F., Skowron, A., Zhong, N. (eds.) RSCTC 2002. LNCS (LNAI), vol. 2475, pp. 405–412. Springer, Heidelberg (2002)
Boros, E., Hammer, P.L., Ibaraki, T., Kogan, A., Mayoraz, E., Muchnik, I.: An Implementation of Logical Analysis of Data. IEEE Transactions on Knowledge and Data Engineering 12, 292–306 (2000)
Pawlak, Z.: Rough Sets. Theoretical Aspects of Reasoning about Data. Kluwer Academic Publishers, Dordrecht (1991)
Słowiński, R. (ed.): Intelligent Decision Support. In: Handbook of Applications and Advances of the Rough Set Theory. Kluwer Academic Publishers, Dordrecht (1992)
Grzymala-Busse, J.W.: LERS — A System for Learning from Examples based on Rough Sets. In: Słowiński, R. (ed.) Intelligent Decision Support, Handbook of Applications and Advances of the Rough Sets Theory, pp. 3–18. Kluwer Academic Publishers, Dordrecht (1992)
Skowron, A.: Extracting Laws from Decision Tables - A Rough Set Approach. Computational Intelligence 11, 371–388 (1995)
Stefanowski, J.: On Rough Set based Approach to Induction of Decision Rules. In: Skowron, A., Polkowski, L. (eds.) Rough Set in Knowledge Discovering, pp. 500–529. Physica Verlag, Heidelberg (1998)
Greco, S., Matarazzo, B., Słowiński, R., Stefanowski, J.: An algorithm for induction of decision rules consistent with the dominance principle. In: Ziarko, W.P., Yao, Y. (eds.) RSCTC 2000. LNCS (LNAI), vol. 2005, pp. 304–313. Springer, Heidelberg (2001)
Freund, Y., Schapire, R.E.: A Decision-theoretic Generalization of On-line Learning and an Application to Boosting. Journal of Computer and System Sciences 55(1), 119–139 (1997)
Hastie, T., Tibshirani, R., Friedman, J.H.: Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, Heidelberg (2003)
Cohen, W.W., Singer, Y.: A Simple, Fast, and Effective Rule Learner. In: Proc. of National Conference on Artificial Intelligence, pp. 335–342 (1999)
Weiss, S.M., Indurkhya, N.: Lightweight Rule Induction. In: Proc. of International Conference on Machine Learning, pp. 1135–1142 (2000)
Friedman, J.H., Popescu, B.E.: Predictive Learning via Rule Ensembles. Annals of Applied Statistics 2(3), 916–954 (2008)
Dembczyński, K., Kotłowski, W., Słowiński, R.: Maximum Likelihood Rule Ensembles. In: Proc. of International Conference on Machine Learning, pp. 224–231 (2008)
Błaszczyński, J., Dembczyński, K., Kotłowski, W., Słowiński, R., Szelag, M.: Ensembles of Decision Rules. Foundations of Computing and Decision Sciences 31(3-4), 221–232 (2006)
Friedman, J.H.: Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics 29(5), 1189–1232 (2001)
Breiman, L.: Bagging Predictors. Machine Learning 24(2), 123–140 (1996)
Mason, L., Baxter, J., Bartlett, P., Frean, M.: Functional Gradient Techniques for Combining Hypotheses. In: Bartlett, P., Schölkopf, B., Schuurmans, D., Smola, A.J. (eds.) Advances in Large Margin Classifiers, pp. 33–58. MIT Press, Cambridge (1999)
Fürnkranz, J.: Rule-based Classification. In: From Local Patterns to Global Models ECML/PKDD 2008 Workshop (2008)
Dembczyński, K., Kotłowski, W., Słowiński, R.: A General Framework for Learning an Ensemble of Decision Rules. In: Fürnkranz, J., Knobbe, A. (eds.) From Local Patterns to Global Models ECML/PKDD 2008 Workshop (2008)
Schapire, R.E., Freund, Y., Bartlett, P., Lee, W.S.: Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods. Annals of Statistics 26(5), 1651–1686 (1998)
Friedman, J.H., Hastie, T., Tibshirani, R.: Additive Logistic Regression: A Statistical View of Boosting. Annals of Statistics (with discussion) 28(2), 337–407 (2000)
Dietterich, T.G.: An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization. Machine Learning 40(2), 139–158 (2000)
Friedman, J.H., Popescu, B.E.: Importance Sampled Learning Ensembles. Research report, Dept. of Statistics, Stanford University (2003)
Dembczyński, K., Kotłowski, W., Słowiński, R.: Solving Regression by Learning an Ensemble of Decision Rules. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2008. LNCS (LNAI), vol. 5097, pp. 533–544. Springer, Heidelberg (2008)
Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)
Asuncion, A., Newman, D.J.: UCI Machine Learning Repository (2007)
Demšar, J.: Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research 7, 1–30 (2006)
Hilderman, R.J., Hamilton, H.J.: Knowledge Discovery and Measures of Interest. Kluwer Academic Publishers, Boston (2001)
Greco, S., Pawlak, Z., Słowiński, R.: Can Bayesian confirmation measures be useful for rough set decision rules? Engineering Applications of Artificial Intelligence 17, 345–361 (2004)
Brzezińska, I., Greco, S., Słowiński, R.: Mining Pareto-optimal Rules with Respect to Support and Anti-support. Engineering Applications of Artificial Intelligence 20, 587–600 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Dembczyński, K., Kotłowski, W., Słowiński, R. (2010). Beyond Sequential Covering – Boosted Decision Rules. In: Koronacki, J., Raś, Z.W., Wierzchoń, S.T., Kacprzyk, J. (eds) Advances in Machine Learning I. Studies in Computational Intelligence, vol 262. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-05177-7_10
Download citation
DOI: https://doi.org/10.1007/978-3-642-05177-7_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-05176-0
Online ISBN: 978-3-642-05177-7
eBook Packages: EngineeringEngineering (R0)