Abstract
A standard classifier is forced to predict the label of every test instance, even when confidence in the predictions is very low. In many scenarios, it would, however, be better to avoid making these predictions, maybe leaving them to a human expert. A classifier with that alternative is referred to as a classifier with reject option. In this paper, we propose an algorithm that, for a particular data set, automatically suggests a number of accuracy levels, which it will be able to meet perfectly, using a classifier with reject option. Since the basis of the suggested algorithm is conformal prediction, it comes with strong validity guarantees. The experimentation, using 25 publicly available two-class data sets, confirms that the algorithm obtains empirical accuracies very close to the requested levels. In addition, in an outright comparison with probabilistic predictors, including models calibrated with Platt scaling, the suggested algorithm clearly outperforms the alternatives.
The authors acknowledge the Swedish Knowledge Foundation, Jönköping University, and the industrial partners for financially supporting the research through the AFAIR project with grant no. 20200223, as part of the research and education environment SPARK at Jönköping University. Helena Löfström is a PhD student in the Industrial Graduate School in Digital Retailing (INSiDR) at the University of Borås, funded by the Swedish Knowledge Foundation, grant no. 20160035.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bache, K., Lichman, M.: UCI machine learning repository (2013)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Chow, C.: On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theory 16(1), 41–46 (1970)
Guan, H., Zhang, Y., Cheng, H.D., Tang, X.: Bounded-abstaining classification for breast tumors in imbalanced ultrasound images. Int. J. Appl. Math. Comput. Sci. 30(2), 325–336 (2020)
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: ICML, pp. 1321–1330. PMLR (2017)
Hamid, K., Asif, A., Abbasi, W., Sabih, D., Minhas, F.U.A.A.: Machine learning with abstention for automated liver disease diagnosis. In: 2017 International Conference on Frontiers of Information Technology (FIT), pp. 356–361 (2017)
Hanczar, B., Dougherty, E.R.: Classification with reject option in gene expression data. Bioinform. 24(17), 1889–1895 (2008)
Herbei, R., Wegkamp, M.H.: Classification with reject option. Can. J. Statist. 34(4), 709–721 (2006)
Johansson, U., Gabrielsson, P.: Are traditional neural networks well-calibrated? In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2019)
Li, M., Sethi, I.K.: Confidence-based classifier design. Pattern Recogn. 39(7), 1230–1240 (2006)
Linusson, H., Johansson, U., Boström, H., Löfström, T.: Classification with reject option using conformal prediction. In: Phung, D., Tseng, V.S., Webb, G.I., Ho, B., Ganji, M., Rashidi, L. (eds.) PAKDD 2018. LNCS (LNAI), vol. 10937, pp. 94–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93034-3_8
Niculescu-Mizil, A., Caruana, R.: Predicting good probabilities with supervised learning. In: ICML, pp. 625–632 (2005)
Papadopoulos, H.: Inductive conformal prediction: theory and application to neural networks. Tools Artif. Intell. 18, 315–330 (2008)
Papadopoulos, H., Proedrou, K., Vovk, V., Gammerman, A.: Inductive confidence machines for regression. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, pp. 345–356. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-36755-1_29
Platt, J.C.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in Large Margin Classifiers, pp. 61–74. MIT Press (1999)
Provost, F., Domingos, P.: Tree induction for probability-based ranking. Mach. Learn. 52(3), 199–215 (2003)
Sayyad Shirabad, J., Menzies, T.: The PROMISE Repository of Software Engineering Databases. University of Ottawa, Canada, School of IT and Engineering (2005)
Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer-Verlag, New York, Inc (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Johansson, U., Löfström, T., Sönströd, C., Löfström, H. (2023). Conformal Prediction for Accuracy Guarantees in Classification with Reject Option. In: Torra, V., Narukawa, Y. (eds) Modeling Decisions for Artificial Intelligence. MDAI 2023. Lecture Notes in Computer Science(), vol 13890. Springer, Cham. https://doi.org/10.1007/978-3-031-33498-6_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-33498-6_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-33497-9
Online ISBN: 978-3-031-33498-6
eBook Packages: Computer ScienceComputer Science (R0)