Abstract
Non-parametric kernel learning (NPKL) methods have been attracted much more attention and achieved outstanding classification performance in the past few years. Their models generally utilize pairwise constraints and are built based on the manifold assumption. But, such an assumption might be invalid for some high-dimensional and sparse data due to the curse of dimensionality, which has a negative influence on the kernel learning performance. In this paper, we try to address this problem using joint dimensionality reduction and kernel learning. Different from traditional approaches which conduct dimensionality reduction and learning tasks in sequence, we propose a novel framework which can seamlessly combine semi-supervised NPKL with dimensionality reduction. Several semi-supervised NPKL algorithms can be derived from this framework, which not only effectively utilize pairwise constraints, but can address the issue of the manifold assumption invalidation. In addition, we apply the proposed method to improve the performance of kernel clustering. Experimental results demonstrate that the proposed method outperforms state-of-the-art semi-supervised NPKL methods and can significantly enhance the performance of kernel clustering.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ashish, S., Patel, V. M., & Rama, C. (2014). Multiple kernel learning for sparse representation-based classification. IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society, 23(7), 3013–3024.
Baghshah, M. S., & Shouraki, S. B. (2010). Kernel-based metric learning for semi-supervised clustering. Neurocomputing, 73(s 7—-9), 1352–1361.
Baghshah, M. S., & Shouraki, S. B. (2011). Learning low-rank kernel matrices for constrained clustering. Neurocomputing, 74(12–13), 2201–2211.
Belkin, M., Niyogi, P., & Sindhwani, V. (2006). Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 7(1), 2399–2434.
Bucak, S. S., & Jain, A. K. (2014). Multiple kernel learning for visual object recognition: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(2), 1–1.
Burer, S., & Monteiro, R. D. C. (2003). A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2), 329–357.
Chen, W., & Feng, G. (2012). Spectral clustering: A semi-supervised approach. Neurocomputing, 77(1), 229–242.
Chen, C., Zhang, J., He, X., & Zhou, Z. H. (2012). Non-parametric kernel learning with robust pairwise constraints. International Journal of Machine Learning and Cybernetics, 3(2), 83–96.
Feiping, N., Zinan, Z., Tsang, I. W., Dong, X., & Changshui, Z. (2011). Spectral embedded clustering: A framework for in-sample and out-of-sample spectral clustering. IEEE Transactions on Neural Networks, 22(11), 1796–1808.
Hoi, S. C. H., Jin, R., & Lyu, M. R. (2007). Learning nonparametric kernel matrices from pairwise constraints. In Proceedings of the 24th International Conference on Machine Learning, (ICML 2007) (pp.361–368).
Hu, E. L., & Kwok, J. T. (2014). Scalable nonparametric low-rank kernel learning using block coordinate descent. IEEE Transactions on Neural Networks and Learning Systems, 26(9), 1927–1938.
Kulis, B. (2009). Low-rank kernel learning with bregman matrix divergences. Journal of Machine Learning Research, 10(1), 341–376.
Lanckriet, G. R. G., Christianini, N., Bartlett, P. L., Ghaoui, L. E., & Jordan, M. I. (2004). Learning the kernel matrix with semi-definite programming. Journal of Machine Learning Research, 5, 27–72.
Li, Z., & Liu, J. (2009). Constrained clustering by spectral kernel learning. In 12th IEEE International Conference on Computer Vision,(ICCV 2009) (pp.421–427). IEEE.
Li, Z., Liu, J., & Tang, X. (2008). Pairwise constraint propagation by semidefinite programming for semi-supervised classification. In International Conference on Machine Learning, (ICML 2008) (pp.576–583).
Liang, Z., Zhang, L., & Liu, J. (2015). A novel multiple kernel learning method based on the kullback–Leibler divergence. Neural Processing Letters, 42(1), 745–762.
Liu, M., Sun, W., & Liu, B. (2015). Multiple kernel dimensionality reduction via spectral regression and trace ratio maximization. Knowledge-Based Systems, 83(1), 159–169.
Liu, B., Xia, S. X., & Zhou, Y. (2013). Unsupervised non-parametric kernel learning algorithm. Knowledge-Based Systems, 44(1), 1–9.
Liwicki, S., Zafeiriou, S. P., & Pantic, M. (2015). Online kernel slow feature analysis for temporal video segmentation and tracking. IEEE Transactions on Image Processing, 24, 2955–2970.
Meng, J., Jung, C., Shen, Y., Jiao, L., & Liu, J. (2015). Adaptive constraint propagation for semi-supervised kernel matrix learning. Neural Processing Letters, 41(1), 1–17.
Nazarpour, A., & Adibi, P. (2015). Two-stage multiple kernel learning for supervised dimensionality reduction. Pattern Recognition, 48(5), 1854–1862.
Orabona, F., Luo, J., & Caputo, B. (2012). Multi kernel learning with online-batch optimization. Journal of Machine Learning Research, 13(1), 227–253.
Peng, J., Zhou, Y., & Chen, C. L. P. (2015). Region-kernel-based support vector machines for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 53(4), 1–15.
Reitmaier, T., & Sick, B. (2015). The responsibility weighted mahalanobis kernel for semi-supervised training of support vector machines for classification. Information Sciences, 323, 179–198.
Shrivastava, A., Pillai, J. K., & Patel, V. M. (2015). Multiple kernel-based dictionary learning for weakly supervised classification. Pattern Recognition, 48(8), 2667–2675.
Xiang, S., Nie, F., & Zhang, C. (2008). Learning a mahalanobis distance metric for data clustering and classification. Pattern Recognition, 41(12), 3600–3612.
Yen-Yu, L., Tyng-Luh, L., & Chiou-Shann, F. (2011). Multiple kernel learning for dimensionality reduction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(6), 1147–1160.
Zhang, X., & Mahoor, M. H. (2015). Task-dependent multi-task multiple kernel learning for facial action unit detection. Pattern Recognition, 51, 187–196.
Zhang, K., Wang, Q., Lan, L., Sun, Y., & Marsic, I. (2014). Sparse semi-supervised learning on low-rank kernel. Neurocomputing, 129(4), 265–272.
Zhong, S., Chen, D., Xu, Q., & Chen, T. (2013). Optimizing the gaussian kernel function with the formulated kernel target alignment criterion for two-class pattern classification. Pattern Recognition, 46(7), 2045–2054.
Zhuang, J., Tsang, I. W., & Hoi, S. C. H. (2011). A family of simple non-parametric kernel learning algorithms. Journal of Machine Learning Research, 12(2), 1313–1347.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (No. 61403394) and the Fundamental Research Funds for the Central Universities (No. 2014QNA46).
Author information
Authors and Affiliations
Corresponding authors
Rights and permissions
About this article
Cite this article
Liu, M., Liu, B., Zhang, C. et al. Embedded non-parametric kernel learning for kernel clustering. Multidim Syst Sign Process 28, 1697–1715 (2017). https://doi.org/10.1007/s11045-016-0440-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11045-016-0440-1