Skip to main content
SpringerLink
Log in

Introduction to Classification Algorithms and Their Performance Analysis Using Medical Examples

  • Chapter
  • First Online:
Data Science for Healthcare

Abstract

In this chapter, we give an introduction to classification algorithms and the metrics that are used to quantify and visualize their performance. We first briefly explain what we mean with a classification algorithm, and, as an example, we describe in more detail the naive Bayesian classification algorithm. Using the concept of a confusion matrix, we next define the various performance metrics that can be derived from it, including sensitivity and specificity that define the two dimensions of ROC space. We next argue that correctly evaluating the performance of a classification algorithm requires taking into account the conditions in which the algorithm has to operate in practice. These so-called operating conditions consist of two elements: class skew and cost skew. We show that both elements can be combined into a single parameter that defines cost, and that iso-cost curves are straight lines in ROC space.

Additionally, as alternatives to ROC space, we briefly review two other spaces, namely, precision-recall space and cost-curve space. The latter was introduced by Drummond and Holte (Explicitly representing expected cost: an alternative to ROC representation. In: Proceedings of the sixth ACM SIGKDD international conference on knowledge discovery and data mining, KDD 2000, Boston, pp 198–207, 2000; Mach Learn 65(1):95–130, 2006). To illustrate the material we present, we will use a number of examples taken from the medical domain.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Chapelle, O., Schölkopf, B., Zien, A. (eds.): Semi-supervised Learning. MIT Press, Cambridge (2006)

    Google Scholar 

  2. Chawla, N.V.: Data mining for imbalanced datasets: an overview. In: Maimon, O., Rokach, L. (eds.) Data Mining and Knowledge Discovery Handbook, Chap. 40, pp. 875–886. Springer, Berlin (2010)

    Google Scholar 

  3. Davis, J., Goadrich, M.: The relationship between precision-recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine Learning, ICML 2006, Pittsburgh, PA, pp. 233–240 (2006)

    Google Scholar 

  4. de Berg, M., van Kreveld, M., Overmars, M., Schwarzkopf, O.: Computational Geometry: Algorithms and Applications. Springer, Berlin (1997)

    Book  Google Scholar 

  5. Dodd, L.E., Pepe, M.S.: Partial AUC estimation and regression. Biometrics 59, 614–623 (2003)

    Article  MathSciNet  Google Scholar 

  6. Domingos, P., Pazzani, M.: Beyond independence: conditions for the optimality of the simple Bayesian classifier. In: Saitta, L. (ed.) Proceedings of the 13th International Conference on Machine Learning, ICML 1996, San Francisco, CA, pp. 105–112 (1996)

    Google Scholar 

  7. Domingos, P., Pazzani, M.: On the optimality of the simple Bayesian classifier under zero-one loss. Mach. Learn. 29(2–3), 103–130 (1997)

    Article  Google Scholar 

  8. Dougherty, J., Kohavi, R., Sahami, M.: Supervised and unsupervised discretization of continuous features. In: Proceedings of the 12th International Conference on Machine Learning, ICML 1995, San Francisco, CA, pp. 194–202 (1995)

    Google Scholar 

  9. Drummond, C., Holte, R.C.: Explicitly representing expected cost: an alternative to ROC representation. In: Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2000, Boston, MA, pp. 198–207 (2000)

    Google Scholar 

  10. Drummond, C., Holte, R.C.: Cost curves: an improved method for visualizing classifier performance. Mach. Learn. 65(1), 95–130 (2006)

    Article  Google Scholar 

  11. Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. Lett. 27, 861–874 (2006)

    Article  Google Scholar 

  12. Flach, P., Hernández-Orallo, J., Ferri, C.: A coherent interpretation of AUC as a measure of aggregated classification performance. In: Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, WA, pp. 657–664 (2011)

    Google Scholar 

  13. Fürnkranz, J., Flach, P.A.: ROC ‘n’ rule learning - towards a better understanding of covering algorithms. Mach. Learn. 58, 39–77 (2005)

    Article  Google Scholar 

  14. Good, I.J.: Probability and the Weighing of Evidence. Griffin, London (1950)

    MATH  Google Scholar 

  15. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)

    MATH  Google Scholar 

  16. Hand, D.J.: Measuring classifier performance: a coherent alternative to the area under the ROC curve. Mach. Learn. 77(1), 103–123 (2009)

    Article  Google Scholar 

  17. Hand, D.J., Till, R.J.: A simple generalization of the area under the ROC curve to multiple class classification problems. Mach. Learn. 45(2), 171–186 (2001)

    Article  Google Scholar 

  18. Hand, D.J., Yu, K.: Idiot’s Bayes – not so stupid after all? Int. Stat. Rev. 69(3), 385–398 (2001)

    MATH  Google Scholar 

  19. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer, Berlin (2008)

    MATH  Google Scholar 

  20. Hernández-Orallo, J.: ROC curves for regression. Pattern Recogn. 46(12), 3395–3411 (2013)

    Article  Google Scholar 

  21. Hernández-Orallo, J., Flach, P.A., Ferri, C.: Brier curves: a new cost-based visualisation of classifier performance. In: Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, WA, pp. 585–592 (2011)

    Google Scholar 

  22. James, G., Witten, D., Hastie, T., Tibshirani, R.: An Introduction to Statistical Learning. Springer, Berlin (2013)

    Book  Google Scholar 

  23. Kononenko, I., Bratko, I., Roškar, E.: Experiments in automatic learning of medical diagnostic rules, technical report, Jozef Stefan Institute, Ljubljana, Yugoslavia (1984)

    Google Scholar 

  24. Kotsiantis, S., Kanellopoulos, D.: Discretization techniques: a recent survey. GESTS Int. Trans. Comput. Sci. Eng. 32(1), 47–58 (2006)

    Google Scholar 

  25. Laird, N.M., Louis, T.A.: Empirical Bayes confidence intervals based on bootstrap samples. J. Am. Stat. Assoc. 82, 739–757 (1987)

    Article  MathSciNet  Google Scholar 

  26. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)

    Article  Google Scholar 

  27. Maron, M.E.: Automatic indexing: an experimental inquiry. J. ACM 8, 404–417 (1960)

    Article  Google Scholar 

  28. McClish, D.K.: Analyzing a portion of the ROC curve. Med. Decis. Making 9, 190–195 (1989)

    Article  Google Scholar 

  29. Millard, L.A.C., Flach, P.A., Higgins, J.P.T.: Rate-constrained ranking and the rate-weighted AUC. In: Proceedings ECML/PKDD. Springer Lecture Notes in Computer Science, Nancy, vol. 8725, pp. 383–398 (2014)

    Google Scholar 

  30. Millard, L.A.C., Kull, M., Flach, P.A.: Rate-oriented point-wise confidence bounds for ROC curves. In: Proceedings ECML/PKDD. Springer Lecture Notes in Computer Science, Nancy, vol. 8725, pp. 404–412 (2014)

    Google Scholar 

  31. Mosteller, F., Tukey, J.W.: Data analysis, including statistics. In: Handbook of Social Psychology. Addison-Wesley, Boston (1968)

    Google Scholar 

  32. Nielsen, T.D., Jensen, F.V.: Bayesian Networks and Decision Graphs. Springer, Berlin (2007)

    MATH  Google Scholar 

  33. Pronk, V., Gutta, S., Verhaegh, W.: Incorporating confidence in a naive Bayesian classifier. In: Ardissono, L., Brna, P., Mitrovic, A. (eds.) Proceedings of the 10th International Conference on User Modeling, Edinburgh. Lecture Notes in Artificial Intelligence, vol. 3538, pp. 317–326. Springer, Berlin (2005)

    Google Scholar 

  34. Pronk, V., Verhaegh, W., Proidl, A., Tiemann, M.: Incorporating user control into recommender systems based on naive Bayesian classification. In: Proceedings of the ACM Conference Series on Recommender Systems, RecSys 2007, Minneapolis, pp. 73–80 (2007)

    Google Scholar 

  35. Provost, F., Fawcett, T.: Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions. In: Proceedings 3rd International Conference on Knowledge Discovery and Data Mining, KDD-97, pp. 43–48. AAAI Press, Newport Beach (1997)

    Google Scholar 

  36. Provost, F., Fawcett, T.: Robust classification for imprecise environments. Mach. Learn. 42(3), 203–231 (2001)

    Article  Google Scholar 

  37. Rish, I.: An empirical study of the naive Bayes classifier. In: Proceedings of the IJCAI-01 Workshop on Empirical Methods in AI, Sicily, Italy, pp. 41–46 (2001)

    Google Scholar 

  38. Silverman, B.W.: Density Estimation for Statistics and Data Analysis. Chapman and Hall, London (1986)

    Book  Google Scholar 

  39. Stone, M.: Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc. 36(2), 111–147 (1974)

    MathSciNet  MATH  Google Scholar 

  40. Weiss, G., Provost, F.: Learning when training data are costly: the effect of class distribution on tree induction. J. Artif. Intell. Res. 19, 315–354 (2003)

    Article  Google Scholar 

  41. Wu, T.-F., Lin, C.-J., Weng, R.C.: Probability estimates for multi-class classification by pairwise coupling. J. Mach. Learn. Res. 5, 975–1005 (2004)

    MathSciNet  MATH  Google Scholar 

  42. Zabell, S.L.: The rule of succession, Erkenntnis [1975−). Bruno de Finetti’s Philos. Probab. 31(2–3), 283–321 (1989)

    Google Scholar 

  43. Zhu, X., Ghahramani, Z., Lafferty, J.: Semi-supervised learning using Gaussian fields and harmonic functions. In: Proceeding of the 20th International Conference on Machine Learning, ICML 2003, Washington, pp. 912–919 (2003)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Philips Research, Eindhoven, The Netherlands

    Jan Korst, Verus Pronk, Mauro Barbieri & Sergio Consoli

Authors
  1. Jan Korst
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Verus Pronk
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mauro Barbieri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sergio Consoli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Verus Pronk .

Editor information

Editors and Affiliations

  1. Philips Research, Eindhoven, The Netherlands

    Sergio Consoli

  2. Dept of Mathematics and Computer Science, University of Cagliari, Cagliari, Italy

    Diego Reforgiato Recupero

  3. Data Science Department, Philips Research, Eindhoven, The Netherlands

    Milan Petković

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Korst, J., Pronk, V., Barbieri, M., Consoli, S. (2019). Introduction to Classification Algorithms and Their Performance Analysis Using Medical Examples. In: Consoli, S., Reforgiato Recupero, D., Petković, M. (eds) Data Science for Healthcare. Springer, Cham. https://doi.org/10.1007/978-3-030-05249-2_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-05249-2_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-05248-5

  • Online ISBN: 978-3-030-05249-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation