Skip to main content
SpringerLink
Log in
Book cover

International Conference on Artificial Intelligence and Soft Computing

ICAISC 2008: Artificial Intelligence and Soft Computing – ICAISC 2008 pp 533–544Cite as

Solving Regression by Learning an Ensemble of Decision Rules

  • Conference paper

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 5097))

Abstract

We introduce a novel decision rule induction algorithm for solving the regression problem. There are only few approaches in which decision rules are applied to this type of prediction problems. The algorithm uses a single decision rule as a base classifier in the ensemble. Forward stagewise additive modeling is used in order to obtain the ensemble of decision rules. We consider two types of loss functions, the squared- and absolute-error loss, that are commonly used in regression problems. The minimization of empirical risk based on these loss functions is performed by two optimization techniques, the gradient boosting and the least angle technique. The main advantage of decision rules is their simplicity and good interpretability. The prediction model in the form of an ensemble of decision rules is powerful, which is shown by results of the experiment presented in the paper.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   189.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Błaszczyński, J., Dembczyński, K., Kotłowski, W., Słowiński, R., Szeląg, M.: Ensemble of decision rules. Foundations of Computing and Decision Sciences (31), 3–4 (2006)

    Google Scholar 

  2. Błaszczyński, J., Dembczyński, K., Kotłowski, W., Słowiński, R., Szeląg, M.: Ensembles of Decision Rules for Solving Binary Classification Problems in the Presence of Missing Values. In: Greco, S., Hata, Y., Hirano, S., Inuiguchi, M., Miyamoto, S., Nguyen, H.S., Słowiński, R. (eds.) RSCTC 2006. LNCS (LNAI), vol. 4259, pp. 224–234. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  3. Boros, E., Hammer, P., Ibaraki, T., Kogan, A., Mayoraz, E., Muchnik, I.: An Implementation of Logical Analysis of Data. IEEE Trans. on Knowledge and Data Engineering 12, 292–306 (2000)

    Article  Google Scholar 

  4. Breiman, L.: Bagging Predictors. Machine Learning 24, 123–140 (1996)

    MathSciNet  MATH  Google Scholar 

  5. Breiman, L.: Arcing classifiers. Annals of Statistics 26, 801–824 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  6. Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and Regression Trees. Wadsworth (1984)

    Google Scholar 

  7. Clark, P., Nibbet, T.: The CN2 induction algorithm. Machine Learning 3, 261–283 (1989)

    Google Scholar 

  8. Cohen, W.: Fast effective rule induction. In: International Conference on Machine Learning, pp. 115–123 (1995)

    Google Scholar 

  9. Cohen, W., Singer, Y.: A simple, fast, and effective rule learner. In: National Conference on Artificial Intelligence, pp. 335–342 (1999)

    Google Scholar 

  10. Demšar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7, 1–30 (2006)

    Google Scholar 

  11. Domingos, P.: Unifying instance-based and rule-based induction. Machine Learning 24, 141–168 (1996)

    Google Scholar 

  12. Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Annals of Statistics 28, 337–407 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  13. Friedman, J.: Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics 29, 1189–1232 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  14. Friedman, J.: Stochastic Gradient Boosting. Computational Statistics & Data Analysis 38, 367–378 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  15. Friedman, J., Popescu, B.: Gradient directed regularization. Technical Report, Dept. of Statistics, Stanford University (2004)

    Google Scholar 

  16. Friedman, J., Popescu, B.: Predictive Learning via Rule Ensembles. Technical Report, Dept. of Statistics, Stanford University (2005)

    Google Scholar 

  17. Freund, Y., Schapire, R.: A decision-theoretic generalization of on-line learning and an application to boosting. J. of Comp. and System Sc. 55, 119–139 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  18. Fürnkranz, J.: Separate-and-conquer rule learning. AI Review 13, 3–54 (1996)

    Google Scholar 

  19. Góra, G., Wojna, A.: RIONA: A New Classification System Combining Rule Induction and Instance-Based Learning. Fundamenta Informaticae 54, 369–390 (2002)

    Google Scholar 

  20. Greco, S., Matarazzo, B., Słowiński, R., Stefanowski, J.: An Algorithm for Induction of Decision Rules Consistent with the Dominance Principle. In: Ziarko, W., Yao, Y. (eds.) RSCTC 2000. LNCS (LNAI), vol. 2005, pp. 304–313. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  21. Grzymala-Busse, J.: LERS — A system for learning from examples based on rough sets. In: [32], pp. 3–18. Kluwer Academic Publishers, Dordrecht (1992)

    Google Scholar 

  22. Hastie, T., Tibshirani, R., Friedman, J.: Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, Heidelberg (2003)

    Google Scholar 

  23. Huber, P.: Robust Estimation of a Location Parameter. Annals of Mathematical Statistics 35, 73–101 (1964)

    Article  MathSciNet  MATH  Google Scholar 

  24. Indurkhya, N., Weiss, S.: Solving Regression Problems with Rule-based Ensemble Classifiers. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 287–292 (2001)

    Google Scholar 

  25. Jovanoski, V., Lavrac, N.: Classification Rule Learning with APRIORI-C. In: Brazdil, P., Jorge, A. (eds.) EPIA 2001. LNCS (LNAI), vol. 2258, pp. 44–51. Springer, Heidelberg (2001)

    Google Scholar 

  26. Mason, L., Baxter, J., Bartlett, P., Frean, M.: Functional gradient techniques for combining hypotheses. In: Advances in Large Margin Classifiers, pp. 33–58. MIT Press, Cambridge (1999)

    Google Scholar 

  27. Michalski, R.: A Theory and Methodology of Inductive Learning. In: Michalski, R., Carbonell, J., Mitchell, T. (eds.) Machine Learning: An Artificial Intelligence Approach, pp. 83–129. Tioga Publishing, Palo Alto (1983)

    Google Scholar 

  28. Pawlak, Z.: Rough Sets. Theoretical Aspects of Reasoning about Data. Kluwer Academic Publishers, Dordrecht (1991)

    MATH  Google Scholar 

  29. Quinlan, J.: Learning with continuous classes. In: Australian Joint Conference on Artificial Intelligence, pp. 343–348 (1992)

    Google Scholar 

  30. Quinlan, J.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Francisco (1993)

    Google Scholar 

  31. Skowron, A.: Extracting laws from decision tables - a rough set approach. Computational Intelligence 11, 371–388 (1995)

    Article  MathSciNet  Google Scholar 

  32. Słowiński, R. (ed.): Intelligent Decision Support. Handbook of Applications and Advances of the Rough Set Theory. Kluwer Academic Publishers, Dordrecht (1992)

    Google Scholar 

  33. Stefanowski, J.: On rough set based approach to induction of decision rules. In: Skowron, A., Polkowski, L. (eds.) Rough Set in Knowledge Discovering, pp. 500–529. Physica Verlag (1998)

    Google Scholar 

  34. Stefanowski, J., Vanderpooten, D.: Induction of decision rules in classification and discovery-oriented perspectives. Int. J. on Intelligent Systems 16, 13–27 (2001)

    Article  MATH  Google Scholar 

  35. Vapnik, V.: The Nature of Statistical Learning Theory, 2nd edn. Springer, Heidelberg (1998)

    Google Scholar 

  36. Weiss, S., Indurkhya, N.: Lightweight rule induction. In: International Conference on Machine Learning, pp. 1135–1142 (2000)

    Google Scholar 

  37. Witten, I., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Computing Science, Poznań University of Technology, 60-965, Poznań, Poland

    Krzysztof Dembczyński, Wojciech Kotłowski & Roman Słowiński

  2. Systems Research Institute, Polish Academy of Sciences, 01-447, Warsaw, Poland

    Roman Słowiński

Authors
  1. Krzysztof Dembczyński
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wojciech Kotłowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Roman Słowiński
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Leszek Rutkowski Ryszard Tadeusiewicz Lotfi A. Zadeh Jacek M. Zurada

Rights and permissions

Reprints and permissions

Copyright information

© 2008 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Dembczyński, K., Kotłowski, W., Słowiński, R. (2008). Solving Regression by Learning an Ensemble of Decision Rules. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing – ICAISC 2008. ICAISC 2008. Lecture Notes in Computer Science(), vol 5097. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-69731-2_52

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-69731-2_52

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-69572-1

  • Online ISBN: 978-3-540-69731-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation