Abstract
In this paper, we propose a kernel ensemble SVM with integrated loss in shared parameters space. Different from the traditional multiple kernel learning methods of seeking linear combinations of basis kernels as a unified kernel, the proposed method aims to find multiple solutions in corresponding Reproducing Kernel Hilbert Spaces (RKHSs) simultaneously. To achieve this goal, we draw on the idea of multi-view data processing, and the individual kernel gram matrix is considered as one view of the data. We, therefore, propose an ensemble idea to combine those multiple individual kernel losses into a whole one through an integrated loss design. Therefore, each model can co-optimize to learn its optimal parameters by minimizing the integrated loss in multiple RKHSs. Besides, another feature of our method is the introduction of shared and specific parameters in multiple RKHSs for learning. In this manner, the proposed model can learn the common and individual structures of the data from its parameters space, thereby improving the accuracy of the classification task and further enhancing the robustness of the proposed ensemble model. Experimental results on several UCI classification and image datasets demonstrate that our method performs best among state-of-the-art MKL methods, such as SimpleMKL, EasyMKL, MREKLM, and MRMKL.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-022-14226-8/MediaObjects/11042_2022_14226_Figa_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-022-14226-8/MediaObjects/11042_2022_14226_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-022-14226-8/MediaObjects/11042_2022_14226_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-022-14226-8/MediaObjects/11042_2022_14226_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11042-022-14226-8/MediaObjects/11042_2022_14226_Fig4_HTML.png)
Similar content being viewed by others
References
Aiolli F, Donini M (2015) Easymkl: a scalable multiple kernel learning algorithm. Neurocomputing 169:215–224
Anstreicher KM (2012) On convex relaxations for quadratically constrained quadratic programming. Math Program 136(2):233–251
Bach FR, Lanckriet GR, Jordan MI (2004) Multiple kernel learning, conic duality, and the smo algorithm. In: Proceedings of the twenty-first international conference on machine learning, p 6
Bach FR, Thibaux R, Jordan MI (2005) Computing regularization paths for learning multiple kernels. Adv Neural Inf Process Syst 17:73–80
Cortes C, Mohri M, Rostamizadeh A (2012) Algorithms for learning kernels based on centered alignment. J Machine Learn Res 13(1):795–828
Crammer K, Keshet J, Singer Y (2002) Kernel design using boosting. Adv Neural Inf Process Syst 15:553–560
Cristianini N, Shawe-Taylor J et al (2000) An introduction to support vector machines and other kernel-based learning methods. Cambridge University Press, Cambridge
Fan Q, Wang Z, Zha H, Gao D (2017) Mreklm: a fast multiple empirical kernel learning machine. Pattern Recogn 61:197–209
Gönen M, Alpaydın E (2011) Multiple kernel learning algorithms. J Machine Learn Res 12:2211–2268
Gönen M, Khan S, Kaski S (2013) Kernelized bayesian matrix factorization. In: International conference on machine learning, PMLR, pp 864–872
Han Y, Yang Y, Li X, Liu Q, Ma Y (2018) Matrix-regularized multiple kernel learning via (r,p) norms. IEEE Trans Neur Netw Learn Syst 29 (10):4997–5007
Han Y, Yang K, Yang Y, Ma Y (2016) Localized multiple kernel learning with dynamical clustering and matrix regularization. IEEE Trans Neural Netw Learn Systs 29(2):486–499
Hinrichs C, Singh V, Peng J, Johnson S (2012) Q-mkl: Matrix-induced regularization in multi-kernel learning with applications to neuroimaging. Adv Neural Inf Process Syst 25:1421–1429
Hu M, Chen Y, Kwok JT-Y (2009) Building sparse multiple-kernel svm classifiers. IEEE Trans Neur Netw 20(5):827–839
Kim TK (2015) T test as a parametric statistic. Korean J Anesthesiol 68(6):540
Kimeldorf G, Wahba G (1971) Some results on tchebycheffian spline functions. J Math Anal Appl 33(1):82–95
Kloft M, Brefeld U, Sonnenburg S, Zien A (2011) Lp-norm multiple kernel learning. J Machine Learn Res 12:953–997
Kwok J-Y, Tsang I-H (2004) The pre-image problem in kernel methods. IEEE Trans Neural Networks 15(6):1517–1525
Lanckriet GR, Cristianini N, Bartlett P, Ghaoui LE, Jordan MI (2004) Learning the kernel matrix with semidefinite programming. J Mach Learn Res 5(Jan):27–72
Lanckriet GR, De Bie T, Cristianini N, Jordan MI, Noble WS (2004) A statistical framework for genomic data fusion. Bioinformatics 20(16):2626–2635
Li Z, Jing X-Y, Zhu X, Zhang H (2017) Heterogeneous defect prediction through multiple kernel learning and ensemble learning. In: 2017 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 91–102
Li J, Li Z, Lu G, Xu Y, Zhang B, Zhang D (2021) Asymmetric gaussian process multi-view learning for visual classification. Inf Fusion 65:108–118
Liu J, Chen Y, Zhang J, Xu Z (2014) Enhancing low-rank subspace clustering by manifold regularization. IEEE Trans Image Process 23(9):4022–4030
Liu H, Wang W, Jiao P (2019) Content based image retrieval via sparse representation and feature fusion. In: 2019 IEEE 8th data driven control and learning systems conference (DDCLS). IEEE, pp 18–23
Mao Q, Tsang IW, Gao S, Wang L (2014) Generalized multiple kernel learning with data-dependent priors. IEEE Transactions on Neural Networks and Learning Systems 26(6):1134–1148
Margineantu DD, Dietterich TG (1997) Pruning adaptive boosting. In: ICML, vol 97, Citeseer, pp 211–218
Ong CS, Smola A, Williamson R et al (2022) Learning the kernel with hyperkernels
Qi C, Zhou Z, Hu L, Wang Q (2016) A framework of multiple kernel ensemble learning for hyperspectral classification. In: 2016 intl IEEE conferences on ubiquitous intelligence & computing, advanced and trusted computing, scalable computing and communications, cloud and big data computing, internet of people, and smart world congress (UIC/ATC/ScalCom/CBDCom/IoP/SmartWorld). IEEE, pp 456–460
Rakotomamonjy A, Bach F, Canu S, Grandvalet Y (2008) Simplemkl. J Machine Learn Res 9:2491–2521
Recht B, Roelofs R, Schmidt L, Shankar V (2022) Do cifar-10 classifiers generalize to cifar-10? arXiv:1806.00451
Scholkopf B, Smola AJ (2001) Learning with kernels: Support vector machines, regularization, optimization and beyond
Shen X-J, Ni C, Wang L, Zha Z-J (2021) Sliker: Sparse loss induced kernel ensemble regression. Pattern Recogn 109:107587
Sonnenburg S, Rätsch G, Schäfer C (2005) A general and efficient multiple kernel learning algorithm. Adv Neural Inf Process Syst 18:1273–1280
Sonnenburg S, Rätsch G, Schäfer C, Schölkopf B (2006) Large scale multiple kernel learning. J Machine Learn Res 7:1531–1565
Sun T, Jiao L, Liu F, Wang S, Feng J (2013) Selective multiple kernel learning for classification with ensemble strategy. Pattern Recogn 46(11):3081–3090
Sun L, Zhao C, Yan Z, Liu P, Duckett T, Stolkin R (2018) A novel weakly-supervised approach for rgb-d-based nuclear waste object detection. IEEE Sensors J 19(9):3487–3500
Varma M, Babu BR (2009) More generality in efficient multiple kernel learning. In: Proceedings of the 26th annual international conference on machine learning, pp 1065–1072
Vishwanathan S, Murty MN (2002) Ssvm: a simple svm algorithm. In: Proceedings of the 2002 international joint conference on neural networks. IJCNN’02 (Cat. No. 02CH37290), vol 3. IEEE, pp 2393–2398
Wang Q, Ding Z, Tao Z, Gao Q, Fu Y (2020) Generative partial multi-view clustering. arXiv:2003.13088
Wu D, Wang B, Precup D, Boulet B (2019) Multiple kernel learning-based transfer regression for electric load forecasting. IEEE Transactions on Smart Grid 11(2):1183–1192
Xia H, Hoi SC (2012) Mkboost: a framework of multiple kernel boosting. IEEE Trans Knowl Data Eng 25(7):1574–1586
Xu X, Tsang IW, Xu D (2013) Soft margin multiple kernel learning. IEEE Trans Neur Netw Learn Syst 24(5):749–761
Ye J, Ji S, Chen J (2020) Multi-class discriminant kernel learning via convex programming. Journal of Machine Learning Research 9(4)
Zheng-Peng W, Zhang X-G (2011) Elastic multiple kernel learning. Acta Automatica Sinica 37(6):693–699
Zhou J, Jiang Z, Chung F-L, Wang S (2022) Formulating ensemble learning of svms into a single svm formulation by negative agreement learning. IEEE Transactions on Systems Man and Cybernetics Systems
Zhou J, Liu J, Luo X (2015) Rademacher complexity bound for domain adaptation regression. In: 2015 conference on technologies and applications of artificial intelligence (TAAI). IEEE, pp 273–280
Funding
This work was funded in part by the Graduate Research and Innovation Projects of Jiangsu Province (No.SJCX21_1694) and the Science and Technology Planning Social Development Project of Zhenjiang City (No.SH2021006).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wu, Y., Shen, XJ., Abhadiomhen, S.E. et al. Kernel ensemble support vector machine with integrated loss in shared parameters space. Multimed Tools Appl 82, 18077–18096 (2023). https://doi.org/10.1007/s11042-022-14226-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-022-14226-8