Abstract
Evaluation of classification performance has been comprehensively studied for both crispy and fuzzy classification tasks. In this paper, we address the hybrid case: evaluating fuzzy prediction results against crispy references. The proposal is motivated by the following facts: (1) most datasets in practice are produced with crispy labels due to the excessive cost of fuzzy labelling; and (2) many state-of-the-art classifiers can yield fuzzy decision values even if they are trained from data with crispy labels. We derive our fuzzy-crispy evaluation criterion based on a widely adopted fuzzy-set-based evaluation method. By exploiting the distribution of decision values, the proposed criterion bears more comprehensive information than conventional crispy classification evaluation criteria. The advantages of the proposed criterion are demonstrated in artificial and real-world classification case studies.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Binaghi, E., Brivio, P.A., Ghezzi, P., Rampini, A.: A fuzzy set-based accuracy assessment of soft classification. Pattern Recogn. Lett. 20(9), 935–948 (1999)
Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 27:1–27:27 (2011). https://doi.org/10.1145/1961189.1961199. http://doi.acm.org/10.1145/1961189.1961199
Dou, W., Ren, Y., Wu, Q., Ruan, S., Chen, Y., Bloyet, D., Constans, J.M.: Fuzzy kappa for the agreement measure of fuzzy classifications. Neurocomputing 70(4–6), 726–734 (2007)
Draper-Gil, G., Lashkari, A.H., Mamun, M.S.I., Ghorbani, A.A.: Characterization of encrypted and VPN traffic using time-related features. In: Proceedings of the 2nd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP, pp. 407–414. INSTICC, SciTePress (2016). https://doi.org/10.5220/0005740704070414
Hagen, A.: Fuzzy set approach to assessing similarity of categorical maps. Int. J. Geogr. Inf. Sci. 17(3), 235–249 (2003)
Jager, G., Benz, U.: Measures of classification accuracy based on fuzzy similarity. IEEE Trans. Geosci. Remote. Sens. 38(3), 1462–1467 (2000)
Keller, J.M., Gray, M.R., Givens, J.A.: A fuzzy k-nearest neighbor algorithm. IEEE Trans. Syst. Man Cybern. 4, 580–585 (1985)
Klir, G.J., Folger, T.A.: Fuzzy Sets, Uncertainty, and Information. Prentice-Hall, Inc., Upper Saddle River (1987)
Lashkari, A.H., Gil, G.D., Mamun, M.S.I., Ghorbani, A.A.: Characterization of tor traffic using time based features. In: Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP, pp. 253–262. INSTICC, SciTePress (2017). https://doi.org/10.5220/0006105602530262
Liu, Y., Zhang, H.H., Wu, Y.: Hard or soft classification large-margin unified machines. J. Am. Stat. Assoc. 106(493), 166–177 (2011)
Sokolova, M., Lapalme, G.: A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 45(4), 427–437 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhu, L., Ban, T., Takahashi, T., Inoue, D. (2018). Employ Decision Values for Soft-Classifier Evaluation with Crispy References. In: Cheng, L., Leung, A., Ozawa, S. (eds) Neural Information Processing. ICONIP 2018. Lecture Notes in Computer Science(), vol 11304. Springer, Cham. https://doi.org/10.1007/978-3-030-04212-7_34
Download citation
DOI: https://doi.org/10.1007/978-3-030-04212-7_34
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04211-0
Online ISBN: 978-3-030-04212-7
eBook Packages: Computer ScienceComputer Science (R0)