Skip to main content
SpringerLink
Log in

Reliability Estimation of Individual Multi-target Regression Predictions

  • Conference paper
  • First Online:
Artificial Life and Computational Intelligence (ACALCI 2017)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10142))

  • 1047 Accesses

Abstract

To estimate the quality of the induced predictive model we generally use measures of averaged prediction accuracy, such as the relative mean squared error on test data. Such evaluation fails to provide local information about reliability of individual predictions, which can be important in risk-sensitive fields (medicine, finance, industry etc.). Related work presented several ways for computing individual prediction reliability estimates for single-target regression models, but has not considered their use with multi-target regression models that predict a vector of independent target variables. In this paper we adapt the existing single-target reliability estimates to multi-target models. In this way we try to design reliability estimates, which can estimate the prediction errors without knowing true prediction errors, for multi-target regression algorithms, as well. We approach this in two ways: by aggregating reliability estimates for individual target components, and by generalizing the existing reliability estimates to higher number of dimensions. The results revealed favorable performance of the reliability estimates that are based on bagging variance and local cross-validation approaches. The results are consistent with the related work in single-target reliability estimates and provide a support for multi-target decision making.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Kukar, M., Kononenko, I.: Reliable classifications with machine learning. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, pp. 219–231. Springer, Heidelberg (2002). doi:10.1007/3-540-36755-1_19

    Chapter  Google Scholar 

  2. Pevec, D., Kononenko, I.: Input dependent prediction intervals for supervised regression. Intell. Data Anal. 18(5), 873–887 (2014)

    Google Scholar 

  3. Bosnić, Z., Kononenko, I.: Comparison of approaches for estimating reliability of individual regression predictions. Data Knowl. Eng. 67(3), 504–516 (2008)

    Article  Google Scholar 

  4. Bosnić, Z., Kononenko, I.: Estimation of individual prediction reliability using the local sensitivity analysis. Appl. Intell. 29(3), 187–203 (2008)

    Article  Google Scholar 

  5. Kocev, D., Džeroski, S., White, M.D., Newell, G.R., Griffioen, P.: Using single-and multi-target regression trees and ensembles to model a compound index of vegetation condition. Ecol. Model. 220(8), 1159–1168 (2009)

    Article  Google Scholar 

  6. Kocev, D., Vens, C., Struyf, J., Džeroski, S.: Ensembles of multi-objective decision trees. In: Kok, J.N., Koronacki, J., Mantaras, R.L., Matwin, S., Mladenič, D., Skowron, A. (eds.) ECML 2007. LNCS (LNAI), vol. 4701, pp. 624–631. Springer, Heidelberg (2007). doi:10.1007/978-3-540-74958-5_61

    Chapter  Google Scholar 

  7. Breiman, L., Friedman, J., Stone, C.J., Olshen, R.A.: Classification and Regression Trees. CRC Press, New York (1984)

    MATH  Google Scholar 

  8. Segal, M.R.: Tree-structured methods for longitudinal data. J. Am. Stat. Assoc. 87(418), 407–418 (1992)

    Article  Google Scholar 

  9. De’Ath, G.: Multivariate regression trees: a new technique for modeling species-environment relationships. Ecology 83(4), 1105–1117 (2002)

    Google Scholar 

  10. Struyf, J., Džeroski, S.: Constraint based induction of multi-objective regression trees. In: Bonchi, F., Boulicaut, J.-F. (eds.) KDID 2005. LNCS, vol. 3933, pp. 222–233. Springer, Heidelberg (2006). doi:10.1007/11733492_13

    Chapter  Google Scholar 

  11. Blockeel, H., De Raedt, L., Ramon, J.: Top-down induction of clustering trees (2000). arXiv preprint: arXiv:cs/0011032

  12. Segal, M., Xiao, Y.: Multivariate random forests. Wiley Interdisc. Rev.: Data Min. Knowl. Discov. 1(1), 80–87 (2011)

    Google Scholar 

  13. Džeroski, S., Ženko, B.: Stacking with multi-response model trees. In: Roli, F., Kittler, J. (eds.) MCS 2002. LNCS, vol. 2364, pp. 201–211. Springer, Heidelberg (2002). doi:10.1007/3-540-45428-4_20

    Chapter  Google Scholar 

  14. Golub, G., Kahan, W.: Calculating the singular values and pseudo-inverse of a matrix. J. Soc. Ind. Appl. Math. Ser. B: Numer. Anal. 2(2), 205–224 (1965)

    Article  MathSciNet  MATH  Google Scholar 

  15. Tsoumakas, G., Katakis, I., Vlahavas, I.: Mining multi-label data. In: Maimon, O., Rokach, L. (eds.) Data Mining and Knowledge Discovery Handbook, pp. 667–685. Springer, New York (2009)

    Chapter  Google Scholar 

  16. Lichman, M.: UCI machine learning repository. School of Information and Computer Sciences, University of California, Irvine (2013). http://archive.ics.uci.edu/ml

  17. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer and Information Science, University of Ljubljana, Večna pot 113, Ljubljana, Slovenia

    Martin Jakomin & Zoran Bosnić

Authors
  1. Martin Jakomin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zoran Bosnić
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martin Jakomin .

Editor information

Editors and Affiliations

  1. The University of Adelaide, Adelaide, South Australia, Australia

    Markus Wagner

  2. RMIT University, Melbourne, Victoria, Australia

    Xiaodong Li

  3. Swinburne University, Melbourne, Victoria, Australia

    Tim Hendtlass

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Jakomin, M., Bosnić, Z. (2017). Reliability Estimation of Individual Multi-target Regression Predictions. In: Wagner, M., Li, X., Hendtlass, T. (eds) Artificial Life and Computational Intelligence. ACALCI 2017. Lecture Notes in Computer Science(), vol 10142. Springer, Cham. https://doi.org/10.1007/978-3-319-51691-2_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-51691-2_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-51690-5

  • Online ISBN: 978-3-319-51691-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation