Abstract
Metric learning aims to transform features of data into another based on some given distance relationships, which may improve the performances of distance-based machine learning models. Most existing methods use the difference between the distance of similar pairs and that of dissimilar pairs as loss functions for training. This kind of loss function may lack interpretability since people can only observe the distance or the difference of the distance, a number with no bounds, but have no idea about how large or small it is. To provide more explanation of these metric learning models, in this paper, we propose the probabilistic theoretical analysis of metric learning, design a special loss function, and propose the Kernelized Probabilistic Metric Learning (KPML) approach. With all the distance values transformed into probabilities, we can, therefore, compare and explain the results of the model. Besides, to effectively make use of both the labeled and unlabeled data to enhance the performance of semi-supervised clustering, we propose a KPML-based approach that leverages metric learning and semi-supervised learning effectively in a novel way. Finally, we use our model to do experiments about kNN-based semi-supervised clustering and the results show that our model significantly outperforms baselines across various datasets.
Keywords
This work was supported by the National Key R&D Program of China [2018YFB1004700]; the National Natural Science Foundation of China [61872238, 61972254]; the Tencent Joint Research Program, and the Open Project Program of Shanghai Key Laboratory of Data Science (No. 2020090600001). The authors also would like to thank Mingding Liao for his contribution on the early version of this paper.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Afzalan, M., Jazizadeh, F.: An automated spectral clustering for multi-scale data. Neurocomputing 347, 94–108 (2019). https://doi.org/10.1016/j.neucom.2019.03.008
Baccour. L., Alimi, A.M, John, R.I: Intuitionistic fuzzy similarity measures and their role in classification. JIIS, 25(2), 221–237 (2016). http://www.degruyter.com/view/j/jisys.2016.25.issue-2/jisys-2015-0086/jisys-2015-0086.xml
Belesiotis, A., Skoutas, D., Efstathiades, C., Kaffes, V., Pfoser, D.: Spatio-textual user matching and clustering based on set similarity joins. Very Large Data Bases J. (VLDBJ) 27(3), 297–320 (2018). https://doi.org/10.1007/s00778-018-0498-5
Bhatnagar, B.L., Singh, S., Arora, C., Jawahar, C.V.: Unsupervised learning of deep feature representation for clustering egocentric actions. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 1447–1453 (2017). https://doi.org/10.24963/ijcai.2017/200
Bohne, J., Ying, Y., Gentric, S., Pontil, M.: Learning local metrics from pairwise similarity data. Pattern Recognit. 75, 315–326 (2018). https://doi.org/10.1016/j.patcog.2017.04.002
Domeniconi, C., Gunopulos, D., Peng, J.: Large margin nearest neighbor classifiers. IEEE Trans. Neural Netw. Learn. Syst. 16(4), 899–909 (2005). https://doi.org/10.1109/TNN.2005.849821
Goldberger, J., Roweis, S.T., Hinton, G.E., Salakhutdinov, R.: Neighbourhood components analysis. In: Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 513–520 (2004). http://papers.nips.cc/paper/2566-neighbourhood-components-analysis
Haponchyk, I., Uva, A., Yu, S., Uryupina, O., Moschitti, A.: Supervised clustering of questions into intents for dialog system applications. In: Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2310–2321 (2018). https://doi.org/10.18653/v1/d18-1254
He, Y., Chen, W., Chen, Y., Mao, Y.: Kernel density metric learning. In: IEEE International Conference on Data Mining (ICDM), pp. 271–280 (2013). https://doi.org/10.1109/ICDM.2013.153
Hsieh, C.K., Yang, L., Cui, Y., Lin, T.Y., Belongie, S., Estrin, D.: Collaborative metric learning. In: International Conference on World Wide Web (WWW), pp. 193–201 (2017). https://doi.org/10.1145/3038912.3052639
Kalintha, W., Ono, S., Numao, M., Fukui, K.: Kernelized evolutionary distance metric learning for semi-supervised clustering. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 4945–4946 (2017). http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14714
Kang, Z., Peng, C., Cheng, Q.: Twin learning for similarity and clustering: a unified kernel approach. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 2080–2086 (2017). http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14569
Kulis, B., et al.: Metric learning: A survey. Found. Trends® Mach. Learn. 5(4), 287–364 (2013). https://doi.org/10.1561/2200000019
Kwok, J.T., Tsang, I.W.: Learning with idealized kernels. In: ACM International Conference on Machine Learning (ICML), pp. 400–407 (2003). http://www.aaai.org/Library/ICML/2003/icml03-054.php
Li, D., Tian, Y.: Global and local metric learning via eigenvectors. Knowl. Based Syst. 116, 152–162 (2017). https://doi.org/10.1016/j.knosys.2016.11.004
Li, X., Yin, H., Zhou, K., Zhou, X.: Semi-supervised clustering with deep metric learning and graph embedding. World Wide Web (WWW) 23(2), 781–798 (2020). https://doi.org/10.1007/s11280-019-00723-8
Li, Y., Tian, X., Tao, D.: Regularized large margin distance metric learning. In: IEEE International Conference on Data Mining (ICDM), pp. 1015–1022 (2016). https://doi.org/10.1109/ICDM.2016.0129
Liu, G., Zheng, K., Wang, Y., Orgun, M.A., Liu, A., Zhao, L., Zhou, X.: Multi-constrained graph pattern matching in large-scale contextual social graphs. In: IEEE International Conference on Data Engineering (ICDE), pp. 351–362 (2015). https://doi.org/10.1109/ICDE.2015.7113297
Nguyen, B., Morell, C., Baets, B.D.: Supervised distance metric learning through maximization of the Jeffrey divergence. Pattern Recognit. 64, 215–225 (2017). https://doi.org/10.1016/j.patcog.2016.11.010
Polat, K.: Similarity-based attribute weighting methods via clustering algorithms in the classification of imbalanced medical datasets. Neural Comput. Appl. 30(3), 987–1013 (2018). https://doi.org/10.1007/s00521-018-3471-8
Schölkopf, B., Smola, A.J.: Learning with Kernels: support vector machines, regularization, optimization, and beyond. In: Adaptive Computation and Machine Learning Series. MIT Press (2002). http://www.worldcat.org/oclc/48970254
Schultz, M., Joachims, T.: Learning a distance metric from relative comparisons. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 41–48 (2003). http://papers.nips.cc/paper/2366-learning-a-distance-metric-from-relative-comparisons
Shalev-Shwartz, S., Singer, Y., Ng, A.Y.: Online and batch learning of pseudo-metrics. In: International Conference on Machine Learning (ICML) (2004). https://doi.org/10.1145/1015330.1015376
Shalev-Shwartz, S., Singer, Y., Ng, A.Y.: Online and batch learning of pseudo-metrics. In: ACM International Conference on Machine Learning (ICML), p. 94 (2004). https://doi.org/10.1145/1015330.1015376
Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press (2004). https://doi.org/10.1017/CBO9780511809682. https://kernelmethods.blogs.bristol.ac.uk/
Simard, P.Y., LeCun, Y., Denker, J.S.: Efficient pattern recognition using a new transformation distance. In: Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 50–58 (1992)
Smola, A.J.: Learning with Kernels. Citeseer (1998). http://d-nb.info/955631580
Song, K., Nie, F., Han, J., Li, X.: Parameter free large margin nearest neighbor for distance metric learning. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 2555–2561 (2017). http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14616
St Amand, J., Huan, J.: Sparse compositional local metric learning. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). pp. 1097–1104 (2017). https://doi.org/10.1145/3097983.3098153
Wang, Q., Yin, H., Hu, Z., Lian, D., Wang, H., Huang, Z.: Neural memory streaming recommender networks with adversarial training. In: ACM SIGKDD International Conference on Knowledge Discovery & Data Mining(KDD), pp. 2467–2475 (2018). https://doi.org/10.1145/3219819.3220004
Weinberger, K.Q., Blitzer, J., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. In: Advances in neural information processing systems (NeurIPS), pp. 1473–1480 (2005). http://papers.nips.cc/paper/2795-distance-metric-learning-for-large-margin-nearest
Weinberger, K.Q., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. J. Mach. Learn. Res. 10, 207–244 (2009). https://dl.acm.org/citation.cfm?id=1577078
Wu, Y., Wang, S., Huang, Q.: Online asymmetric similarity learning for cross-modal retrieval. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 3984–3993 (2017). https://doi.org/10.1109/CVPR.2017.424
Xing, E.P., Jordan, M.I., Russell, S.J., Ng, A.Y.: Distance metric learning with application to clustering with side-information. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 505–512 (2002). http://papers.nips.cc/paper/2164-distance-metric-learning-with-application-to-clustering
Ye, H.J., Zhan, D.C., Si, X.M., Jiang, Y.: Learning mahalanobis distance metric: Considering instance disturbance helps. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 3315–3321 (2017). https://doi.org/10.24963/ijcai.2017/463
Yu, S., Chu, S.W., Wang, C., Chan, Y., Chang, T.: Two improved k-means algorithms. Appl. Soft Comput. 68, 747–755 (2018). https://doi.org/10.1016/j.asoc.2017.08.032
Zhai, D., Liu, X., Chang, H., Zhen, Y., Chen, X., Guo, M., Gao, W.: Parametric local multiview hamming distance metric learning. Pattern Recognit. 75, 250–262 (2018). https://doi.org/10.1016/j.patcog.2017.06.018
Zhang, J., Zhang, L.: Efficient stochastic optimization for low-rank distance metric learning. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 933–940 (2017). http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14373
Zheng, K., Zheng, Y., Yuan, N.J., Shang, S., Zhou, X.: Online discovery of gathering patterns over trajectories. IEEE Trans. Knowl. Data Eng. 26(8), 1974–1988 (2014). https://doi.org/10.1109/TKDE.2013.160
Zhu, H., Long, M., Wang, J., Cao, Y.: Deep hashing network for efficient similarity retrieval. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 2415–2421 (2016). http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12039
Zuo, W., Wang, F., Zhang, D., Lin, L., Huang, Y., Meng, D., Zhang, L.: Distance metric learning via iterated support vector machines. IEEE Trans. Image Process. 26(10), 4937–4950 (2017). https://doi.org/10.1109/TIP.2017.2725578
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, C., Hu, Y., Gao, X., Chen, G. (2020). KPML: A Novel Probabilistic Perspective Kernel Mahalanobis Distance Metric Learning Model for Semi-supervised Clustering. In: Hartmann, S., Küng, J., Kotsis, G., Tjoa, A.M., Khalil, I. (eds) Database and Expert Systems Applications. DEXA 2020. Lecture Notes in Computer Science(), vol 12392. Springer, Cham. https://doi.org/10.1007/978-3-030-59051-2_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-59051-2_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59050-5
Online ISBN: 978-3-030-59051-2
eBook Packages: Computer ScienceComputer Science (R0)