Abstract
For semi-supervised learning, we propose the Laplacian embedded multiple kernel regression model. As we incorporate the multiple kernel occasion into manifold regularization framework, the models we proposed are flexible in many kinds of datasets and have a solid theoretical foundation. The proposed model can solve the two problems, which are the computation cost of manifold regularization framework and the difficulty in dealing with multi-source or multi-attribute datasets. Though manifold regularization is a convex optimization formulation, it often leads to dense matrix inversion with computation cost. Laplacian embedded method we adopted can solve the problem, however it lacks the proper ability to process complex datasets. Therefore, we further use multiple kernel learning as a part of the proposed model to strengthen its ability. Experiments on several datasets compared with the state-of-the-art methods show the effectiveness and efficiency of the proposed model.
Similar content being viewed by others
References
Zhu, X.: Semi-supervised learning literature survey. Comput. Sci. 37(1), 63–77 (2005)
Zhu, S., Sun, X., Jin, D.: Multi-view semi-supervised learning for image classification. Neurocomputing 208, 136–142 (2016)
Amini, M.R., Usunier, N.: Semi-supervised learning. Intell. Syst. Ref. Libr. 49(2), 215–239 (2015)
Mehrkanoon, S., Alzate, C., Mall, R., et al.: Multiclass semisupervised learning based upon kernel spectral clustering. IEEE Trans. Neural Netw. Learn. Syst. 26(4), 720–733 (2015)
Lu, Z., Wang, L.: Noise-robust semi-supervised learning via fast sparse coding. Pattern Recognit. 48(2), 605–612 (2015)
Chapelle, O., Weston, J., Schölkopf, B.: Cluster kernels for semi supervised learning. Adv. Neural Inf. Process. Syst. 585–592 (2002)
Chapelle, O., Schölkopf, B., Zien, A.: Semi-supervised learning. MIT Press, Cambridge (2006)
Joachims, T.: Transductive inference for text classification using support vector machines. Proc. Int. Conf. Mach. Learn. 99, 200–209 (1999)
Bennett, K., Demiriz, A.: Semi-supervised support vector machines. Adv. Neural Inf. Process. Syst. 368–374 (1999)
Chapelle, O., Zien, A.: Semi-supervised classification by low density separation. Proc. Int. Workshop Artif. Intell. Stat. 57–64 (2005)
Yanai, K.: Tools on support vector machines: SVMLight, LIBSVM, SHOGUN. J. Inst. Image Inf. Telev. Eng. 63, 1778–1781 (2009)
Sindhwani, V., Keerthi, S.S., Chapelle, O.: Deterministic annealing for semi-supervised kernel machines. In: Proceedings of the International Conference on Machine Learning. ACM, pp. 841–848 (2006)
Collobert, R., Sinz, F., Weston, J., et al.: Large scale transductive SVMs. J. Mach. Learn. Res. 7, 1687–1712 (2006)
Zhao, H.: Combining labeled and unlabeled data with graph embedding. Neurocomputing 69(16), 2385–2389 (2006)
Zhu, X., Ghahramani, Z., Lafferty, J.: Semi-supervised learning using gaussian fields and harmonic functions. Proc. Int. Conf. Mach. Learn. 3, 912–919 (2003)
Zhou, D., Bousquet, O., Lal, T.N., et al.: Learning with local and global consistency. Adv. Neural Inf. Process. Syst. 16, 321–328 (2003)
Belkin, M., Niyogi, P., Sindhwani, V.: Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J. Mach. Learn. Res. 7, 2399–2434 (2006)
Chen, L., Tsang, I.W., Xu, D.: Laplacian embedded regression for scalable manifold regularization. IEEE Trans. Neural Netw. Learn. Syst. 23(6), 902–915 (2012)
Zhang, K., Wang, Q., Lan, L., et al.: Sparse semi-supervised learning on low-rank kernel. Neurocomputing 129(4), 265–272 (2014)
Zhang, Z., Zhang, Y., Li, F.Z., et al.: Discriminative sparse flexible manifold embedding with novel graph for robust visual representation and label propagation. Pattern Recognit. 61(1), 492–510 (2017)
Yu, J., Rui, Y., Chen, B.: Exploiting click constraints and multi-view features for image re-ranking. IEEE Trans. Multimed. 16(1), 159–1682 (2013)
Yu, J., Tao, D., Rui, Y., et al.: Pairwise constraints based multiview features fusion for scene classification. Pattern Recognit. 46(2), 483–496 (2013)
Zhou, D., Huang, J., Schölkopf, B.: Learning with hypergraphs: clustering, classification, and embedding. Adv. Neural Inf. Process. Syst. 19, 1601–1608 (2007)
Paulsen, V.I., Raghupathi, M.: An Introduction to the Theory of Reproducing Kernel Hilbert Spaces. Cambridge University Press, Cambridge (2016)
Gönen, M., Alpaydın, E.: Multiple kernel learning algorithms. J. Mach. Learn. Res. 12, 2211–2268 (2011)
Lanckriet, G.R.G., Cristianini, N., Bartlett, P., et al.: Learning the kernel matrix with semidefinite programming. J. Mach. Learn. Res. 5, 27–72 (2004)
Zare, T., Sadeghi, M.T.: A novel multiple kernel-based dictionary learning for distributive and collective sparse representation based classifiers. Neurocomputing 234, 164–173 (2017)
Rakotomamonjy, A., Bach, F.R., Canu, S., et al.: SimpleMKL. J. Mach. Learn. Res. 9, 2491–2521 (2008)
Lu, M., Yang, L., Wang, J., et al.: Applications of pointgroup density cartography based on kernel density estimation. Eng. Surv. Mapp. (2017)
Duan, L., Tsang, I.W., Xu, D.: Domain transfer multiple kernel learning. IEEE Trans. Pattern Anal. Mach. Intell. 34(3), 465–479 (2011)
Kumar A, Niculescu-Mizil A, Kavukcuoglu K, et al. A binary classification framework for two-stage multiple kernel learning[J]. arXiv preprint arXiv:1206.6428, 2012
Suzuki, T., Tomioka, R.: SpicyMKL: a fast algorithm for multiple kernel learning with thousands of kernels. Mach. Learn. 85(1–2), 77–108 (2011)
Cortes, C., Mohri, M., Rostamizadeh, A.: Learning non-linear combinations of kernels. Adv. Neural Inf. Process. Syst. 396–404 (2009)
Kloft, M., Brefeld, U., Sonnenburg, S., et al.: Non-sparse regularization and efficient training with multiple kernels. (2010). arXiv:1003.0079
Jain, A., Vishwanathan, S.V.N., Varma, M.: SPF-GMKL generalized multiple kernel learning with a million kernels. In: Proceedings of the International Conference on Knowledge Discovery and Data Mining. ACM, pp. 750-758 (2012)
Cortes, C., Mohri, M., Rostamizadeh, A.: Generalization bounds for learning kernels. Proc. Int. Conf. Mach. Learn. pp. 247–254 (2010)
Cortes, C., Kloft, M., Mohri, M.: Learning kernels using local Rademacher complexity. Adv. Neural Inf. Process. Syst. 2760–2768 (2013)
Schölkopf, B., Herbrich, R., Smola, A.J.: A generalized representer theorem, pp. 416–426. In: Computational learning theory. Springer, Berlin (2001)
Yufeng, L., James, T.K., Zhihua, Z.: Semi-supervised learning using label mean. In: Proceedings of the International Conference on Machine Learning. ACM, pp. 633–640 (2009)
Bache, K., Lichman, M.: UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine (2013)
Griffin, G., Holub, A., Perona, P.: Caltech-256 Object Category Dataset. California Institute of Technology, Pasadena (2007)
Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 27 (2011)
Acknowledgements
This work is supported by Fundamental Research Funds for the Central Universities (No. FRF-TP-16-082A1).
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Full name | Abbreviation | Full name | Abbreviation |
---|---|---|---|
Semi-supervised learning | SSL | Support vector machine | SVM |
Semi-supervised support vector machine | S\(^{3}\)VM | Mean Semi-supervised support vector machine | MeanS\(^{3}\)VM |
Gaussian random field | GRF | K-nearest neighbor | KNN |
Low-density separation | LDS | Deterministic annealing | DA |
Laplacian support vector machine | LapSVM | Manifold regularization | MR |
Reproducing kernel Hilbert space | RKHS | Laplacian embedded support vector regression | LapESVR |
Laplacian embedded multiple kernel regression | LapEMKR | Multiple kernel learning | MKL |
Rights and permissions
About this article
Cite this article
Yang, T., Fu, D. & Li, X. Semi-supervised classification of multiple kernels embedding manifold information. Cluster Comput 20, 3417–3426 (2017). https://doi.org/10.1007/s10586-017-1123-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10586-017-1123-x