Skip to main content
SpringerLink
Log in

An element-wise kernel learning framework

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

An effective kernel learning framework is a fundamental issue which has been attracted considerable attention during the past decade. However, existing multiple kernel learning algorithms follow the assumption that an optimal kernel is a weighted combination of pre-specified kernels, leading to limited kernel representation and insufficient flexibility. Moreover, data-dependent kernel learning approaches explore a flexible kernel matrix in the neighborhood area of the fixed initial kernel matrix, resulting in the restriction on the kernel search space. To solve these limitations, we propose element-wise kernel learning via the connection between representative kernel learning and parameter-free kernel learning. A data-adaptive kernel matrix without any specific formulations is imposed on the representative kernels in an element-wise manner. To diminish the adverse effect of the correlated information among pre-specified base kernels, representative kernels are diversely determined by the kernel selection process. The extensive experiments on benchmark and real-world datasets indicate our proposed framework achieves superior performance against well-known kernel-based algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. https://archive.ics.uci.edu/ml/datasets.html

  2. https://github.com/csliangdu/RMKKM/tree/master/data

  3. https://lig-membres.imag.fr/grimal/data.html

  4. https://github.com/KunyuLin/Multi-view-Datasets

  5. https://www.robots.ox.ac.uk/~vgg/data/flowers/17/

  6. https://archive.ics.uci.edu/ml/machine-learning-databases/mfeat/

  7. http://www.ee.columbia.edu/ln/dvmm/CCV/

  8. https://github.com/mehmetgonen/bemkl

  9. https://github.com/ericstrobl/deepMKL

  10. https://github.com/amansinha/learning-kernels

  11. http://www.lfhsgre.org/selected%20publications/

References

  1. Scholkopf B, Smola AJ (2018) Learning with kernels: Support vector machines, Regularization, Optimization, and Beyond. Adaptive Computation and Machine Learning series ???

  2. Vapnik V (2013) The nature of statistical learning theory. Springer, Berlin

    MATH  Google Scholar 

  3. Kafai M, Eshghi K (2017) Croification: accurate kernel classification with the efficiency of sparse linear svm. IEEE Trans Pattern Anal Mach Intell 41(1):34–48

    Article  Google Scholar 

  4. Takeda H, Farsiu S, Milanfar P (2007) Kernel regression for image processing and reconstruction. IEEE Trans Image process 16(2):349–366

    Article  MathSciNet  Google Scholar 

  5. Liu X, Zhou S, Wang Y, Li M, Dou Y, Zhu E, Yin J (2017) Optimal neighborhood kernel clustering with multiple kernels. In: Proceedings of the AAAI conference on artificial intelligence, vol 31, pp 2266–2272

  6. Liu X, Zhu X, Li M, Wang L, Zhu E, Liu T, Kloft M, Shen D, Yin J, Gao W (2019) Multiple kernel k k-means with incomplete kernels. IEEE Trans Pattern Anal Mach Intell 42(5):1191–1204

    Google Scholar 

  7. Wang S, Liu X, Liu L, Zhou S, Zhu E (2021) Late fusion multiple kernel clustering with proxy graph refinement. IEEE Transactions on Neural Networks and Learning Systems

  8. Xia G, Chen B, Sun H, Liu Q (2020) Nonconvex low-rank kernel sparse subspace learning for keyframe extraction and motion segmentation. IEEE Trans Neural Netw Learn Syst 32(4):1612–1626

    Article  MathSciNet  Google Scholar 

  9. Gu Y, Wang C, You D, Zhang Y, Wang S, Zhang Y (2012) Representative multiple kernel learning for classification in hyperspectral imagery. IEEE Trans Geosci Remote Sens 50(7):2852–2865

    Article  Google Scholar 

  10. Poria S, Chaturvedi I, Cambria E, Hussain A (2016) Convolutional mkl based multimodal emotion recognition and sentiment analysis. In: 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, pp 439–448

  11. Zhu L, Li W, Shi Y, Guo K (2020) Sentivec: Learning sentiment-context vector via kernel optimization function for sentiment analysis. IEEE Trans Neural Netw Learn Syst 32(6):2561–2572

    Article  Google Scholar 

  12. Poria S, Peng H, Hussain A, Howard N, Cambria E (2017) Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing 261:217–230

    Article  Google Scholar 

  13. Manica M, Cadow J, Mathis R, Martinez MR (2019) Pimkl: Pathway-induced multiple kernel learning. NPJ Syst Biol Appl 5(1):1–8

    Article  Google Scholar 

  14. Ye J, Chen K, Wu T, Li J, Zhao Z, Patel R, Bae M, Janardan R, Liu H, Alexander G et al (2008) Heterogeneous data fusion for alzheimer’s disease study. In: Proceedings of the 14th ACM SIGKDD International conference on knowledge discovery and data mining, pp 1025–1033

  15. Wang H, Wang Q, Li P, Zuo W (2021) Multi-scale structural kernel representation for object detection, vol 110

  16. Gönen M, Alpaydın E (2011) Multiple kernel learning algorithms. J Mach Learn Res 12:2211–2268

    MathSciNet  MATH  Google Scholar 

  17. Cristianini N, Kandola J, Elisseeff A, Shawe-Taylor J (2006) On kernel target alignment. In: Innovations in machine learning. Springer, pp 205–256

  18. Cortes C, Mohri M, Rostamizadeh A (2012) Algorithms for learning kernels based on centered alignment. J Mach Learn Res 13(1):795–828

    MathSciNet  MATH  Google Scholar 

  19. Wang T, Tian S, Huang H, Deng D (2009) Learning by local kernel polarization. Neurocomputing 72(13-15):3077–3084

    Article  Google Scholar 

  20. Nguyen CH, Ho TB (2008) An efficient kernel matrix evaluation measure. Pattern Recogn 41(11):3366–3372

    Article  MATH  Google Scholar 

  21. Ying Y, Huang K, Campbell C (2009) Enhanced protein fold recognition through a novel data integration approach. BMC bioinformatics 10(1):1–18

    Article  Google Scholar 

  22. Lanckriet GR, Cristianini N, Bartlett P, Ghaoui LE, Jordan MI (2004) Learning the kernel matrix with semidefinite programming. J Mach Learn Res 5(Jan):27–72

    MathSciNet  MATH  Google Scholar 

  23. Kandola J, Shawe-Taylor J, Cristianini N (2002) Optimizing kernel alignment over combinations of kernel

  24. Cortes C, Mohri M, Rostamizadeh A (2010) Two-stage learning kernel algorithms

  25. Liu J, Liu X, Xiong J, Liao Q, Zhou S, Wang S, Yang Y (2020) Optimal neighborhood multiple kernel clustering with adaptive local kernels. IEEE Transactions on Knowledge and Data Engineering

  26. Liu Y, Liao S, Lin H, Yue Y, Wang W (2017) Infinite kernel learning: generalization bounds and algorithms. In: Proceedings of the AAAI Conference on artificial intelligence, vol 31, pp 2280–2286

  27. Yao Y, Li Y, Jiang B, Chen H (2020) Multiple kernel k-means clustering by selecting representative kernels. IEEE Transactions on Neural Networks and Learning Systems

  28. Li F, Fu Y, Dai Y-H, Sminchisescu C, Wang J (2009) Kernel learning by unconstrained optimization. In: Artificial intelligence and statistics. PMLR, pp 328–335

  29. Mao Q, Tsang IW (2010) Parameter-free spectral kernel learning. In: Proceedings of the Twenty-Sixth conference on uncertainty in artificial intelligence, pp 350–357

  30. Kulis B, Sustik MA, Dhillon IS (2009) Low-rank kernel learning with bregman matrix divergences. Journal of Machine Learning Research 10(2)

  31. Jain P, Kulis B, Davis JV, Dhillon IS (2012) Metric and kernel learning using a linear transformation. J Mach Learn Res 13(1):519–547

    MathSciNet  MATH  Google Scholar 

  32. Pan B, Chen W-S, Xu C, Chen B (2015) A novel framework for learning geometry-aware kernels. IEEE Trans Neural Netw Learn Syst 27(5):939–951

    Article  MathSciNet  Google Scholar 

  33. Lu Z, Jain P, Dhillon IS (2009) Geometry-aware metric learning. In: Proceedings of the 26th Annual international conference on machine learning, pp 673–680

  34. Liu F, Huang X, Gong C, Yang J, Li L (2018) Nonlinear pairwise layer and its training for kernel learning. In: AAAI, pp 3659– 3666

  35. Liu F, Huang X, Gong C, Yang J, Li L (2020) Learning data-adaptive non-parametric kernels. J Mach Learn Res 21(208):1–39

    MathSciNet  MATH  Google Scholar 

  36. Alavi F, Hashemi S (2020) Fkndt: A flexible kernel by negotiating between data-dependent kernel learning and task-dependent kernel learning. In: 2020 25th International Computer Conference, Computer Society of Iran (CSICC). IEEE, pp 1–7

  37. Argyriou A, Micchelli CA, Pontil M (2005) Learning convex combinations of continuously parameterized basic kernels. In: International conference on computational learning theory. Springer, pp 338–352

  38. Cortes C, Mohri M, Rostamizadeh A (2009) L2 regularization for learning kernels. In: Proceedings of the Twenty-Fifth Conference on uncertainty in artificial intelligenc, pp 109–116

  39. Kloft M, Brefeld U, Sonnenburg S, Zien A (2011) Lp-norm multiple kernel learning. J Mach Learn Res 12:953–997

    MathSciNet  MATH  Google Scholar 

  40. Damoulas T, Girolami MA (2008) Probabilistic multi-class multi-kernel learning: on protein fold recognition and remote homology detection. Bioinformatics 24(10):1264–1270

    Article  Google Scholar 

  41. Zhang Z, Dai G, Jordan MI (2011) Bayesian generalized kernel mixed models. J Mach Learn Res 12:111–139

    MathSciNet  MATH  Google Scholar 

  42. Gonen M (2012) Bayesian efficient multiple kernel learning. Proceedings of the 29th International Coference on International Conference on Machine Learning, pp 91–98

  43. Zhuang J, Tsang IW, Hoi SC (2011) Two-layer multiple kernel learning. In: Proceedings of the Fourteenth international conference on artificial intelligence and statistics, pp 909– 917

  44. Strobl EV, Visweswaran S (2013) Deep multiple kernel learning. In: 2013 12th International conference on machine learning and applications. IEEE, vol 1, pp 414–417

  45. Wilson AG, Hu Z, Salakhutdinov R, Xing EP (2016) Deep kernel learning. In: Artificial intelligence and statistics, pp 370–378

  46. Bohn B, Rieger C, Griebel M (2019) A representer theorem for deep kernel learning. J Mach Learn Res 20:64–1

    MathSciNet  MATH  Google Scholar 

  47. Chapelle O, Vapnik V, Bousquet O, Mukherjee S (2002) Choosing multiple parameters for support vector machines. Mach Learn 46(1-3):131–159

    Article  MATH  Google Scholar 

  48. Cortes C, Kloft M, Mohri M (2013) Learning kernels using local rademacher complexity. Advances in neural information processing systems 26:2760–2768

    Google Scholar 

  49. Liu Y, Liao S (2015) Eigenvalues ratio for kernel selection of kernel methods. In: Proceedings of the AAAI Conference on artificial intelligence, vol 29, pp 2814–2820

  50. Liu X, Yin J, Wang L, Liu L, Liu J, Hou C, Zhang J (2012) An adaptive approach to learning optimal neighborhood kernels. IEEE Trans Cybern 43(1):371–384

    Google Scholar 

  51. Kang Z, Wen L, Chen W, Xu Z (2019) Low-rank kernel learning for graph-based clustering. Knowl-Based Syst 163:510–517

    Article  Google Scholar 

  52. Zhang X, Sun H, Liu Z, Ren Z, Cui Q, Li Y (2019) Robust low-rank kernel multi-view subspace clustering based on the schatten p-norm and correntropy. Inf Sci 477:430–447

    Article  MATH  Google Scholar 

  53. Boyd S, Parikh N, Chu E (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers Now Publishers Inc???

  54. Fanuel M, Aspeel A, Delvenne J-C, Suykens JA (2022) Positive semi-definite embedding for dimensionality reduction and out-of-sample extensions. SIAM J Math Data Sci 4(1):153– 178

    Article  MathSciNet  MATH  Google Scholar 

  55. Pan B, Chen W-S, Chen B, Xu C, Lai J (2016) Out-of-sample extensions for non-parametric kernel methods. IEEE Trans Neural Netw Learn Syst 28(2):334–345

    Article  Google Scholar 

  56. Zhou Q, Zhao Q (2015) Flexible clustered multi-task learning by learning representative tasks. IEEE Trans Pattern Anal Mach Intell 38(2):266–278

    Article  Google Scholar 

  57. Elhamifar E, Sapiro G, Sastry SS (2015) Dissimilarity-based sparse subset selection. IEEE Trans Pattern Anal Mach Intell 38(11):2182–2197

    Article  Google Scholar 

  58. Yao Y, Cao J, Chen H (2019) Robust task grouping with representative tasks for clustered multi-task learning. In: Proceedings of the 25th ACM SIGKDD International conference on knowledge discovery & data mining, pp 1408–1417

  59. Boyd S, Boyd SP, Vandenberghe L (2004) Convex Optimization Cambridge university press???

  60. Henrion D, Malick J (2012) Projection methods in conic optimization. In: Handbook on semidefinite, conic and polynomial optimization. Springer??? pp 565–600

  61. Platt J (1998) Sequential minimal optimization: A fast algorithm for training support vector machines

  62. Goldfarb D, Liu S (1990) An o (n 3 l) primal interior point algorithm for convex quadratic programming. Math Program 49(1):325–340

    Article  MathSciNet  MATH  Google Scholar 

  63. Lee J, Balakrishnan V, Koh C-K, Jiao D (2009) From o (k 2 n) to o (n): A fast complex-valued eigenvalue solver for large-scale on-chip interconnect analysis. In: 2009 IEEE MTT-S International microwave symposium digest. IEEE, pp 181–184

  64. Liu X, Wang L, Zhu X, Li M, Zhu E, Liu T, Liu L, Dou Y, Yin J (2019) Absent multiple kernel learning algorithms. IEEE Trans Pattern Anal Mach Intell 42(6):1303–1316

    Article  Google Scholar 

  65. Sinha A, Duchi JC (2016) Learning kernels with random features. Advances in Neural Information Processing Systems 29

  66. Zhou S, Liu X, Li M, Zhu E, Liu L, Zhang C, Yin J (2019) Multiple kernel clustering with neighbor-kernel subspace segmentation. IEEE Trans Neural Netw Learn Syst 31(4):1351–1362

    Article  MathSciNet  Google Scholar 

  67. Yu Y, Ji Z, Guo J, Pang Y (2017) Transductive zero-shot learning with adaptive structural embedding. IEEE Trans Neural Netw Learn Syst 29(9):4116–4127

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. CSE and IT Dept., Shiraz University, Mollasadra Ave., Shiraz, 71348 51154, Fars, Iran

    Fatemeh Alavi & Sattar Hashemi

Authors
  1. Fatemeh Alavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sattar Hashemi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fatemeh Alavi.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alavi, F., Hashemi, S. An element-wise kernel learning framework. Appl Intell 53, 9531–9547 (2023). https://doi.org/10.1007/s10489-022-04020-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04020-2

Keywords

Search

Navigation