Skip to main content
SpringerLink
Log in

Parallel multiple kernel learning: a hybrid alternating direction method of multipliers

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Multiple kernel learning (MKL) has recently become a hot topic in kernel methods. However, many MKL algorithms suffer from high computational cost. Moreover, standard MKL algorithms face the challenge of the rapid development of distributed computational environment such as cloud computing. In this study, a framework for parallel multiple kernel learning (PMKL) using hybrid alternating direction method of multipliers (H-ADMM) is developed to integrate the MKL algorithms and the multiprocessor system. The global problem with multiple kernel is divided into multiple local problems each of which is optimized in a local processor with a single kernel. An H-ADMM is proposed to make the local processors coordinate with each other to achieve the global optimal solution. The results of computational experiments show that PMKL exhibits high classification accuracy and fast computational speed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Vapnik VN (1998) Statistic learning theory. Wiley, New York

    Google Scholar 

  2. Scholkopf B, Smolla A (2002) Learning with kernels-support vector machines, regularization, optimization and beyond. MIT press, Cambridge

    Google Scholar 

  3. Parrado-Hernandez E, Arenas-Garcia J, Mora-Jimenez I et al (2003) On problem-oriented kernel refining. Neurocomputing 55:135–150

    Article  Google Scholar 

  4. Abbasnejad M, Ramachandram D, Mandava R (2012) A survey of the state of the art in learning the kernels. Knowl Inf Syst 31:193–221

    Article  Google Scholar 

  5. Gonen M, Alpaydın E (2011) Multiple kernel learning algorithms. J Mach Learn Res 12:2211–2268

    Google Scholar 

  6. Chapelle O, Vapnik VN, Bousquet O et al (2002) Choosing multiple parameters for support vector machines. Mach Learn 46:131–159

    Article  MATH  Google Scholar 

  7. Gunn SR, Kandola JS (2002) Structural modeling with sparse kernels. Mach Learn 48:137–163

    Article  MATH  Google Scholar 

  8. Bi JB, Zhang T, Bennett KP (2004) Column-generation boosting methods for mixture of kernels. In: Proceedings of the 10th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, NY, USA, pp 521–526

  9. Bach FR, Lanckrient GRG, Jordan MI (2004) Multiple kernel learning, conic duality and the SMO algorithm. In: Russell G, Dale S (eds) Proceedings of the twenty first international conference on machine learning. ACM, New York, NY, USA, pp 41–48

  10. Lanckrient GRG, Cristianini N, Bartlett P et al (2004) Learning the kernel matrix with semidefinite programming. J Mach Learn Res 5:527–572

    Google Scholar 

  11. Lanckriet GRG, Bie TD, Cristianini N et al (2004) A statistical framework for genomic data fusion. Bioinformatics 20:2626–2635

    Article  Google Scholar 

  12. Sonnenburg S, Ratsch G, Schafer C et al (2006) Large scale multiple kernel learning. J Mach Learn Res 1:1–18

    MathSciNet  Google Scholar 

  13. Chen ZY, Li JP, Wei LW (2007) A multiple kernel support vector machine scheme for feature selection and rule extraction from gene expression data of cancer tissue. Artif Intell Med 41:161–175

    Article  Google Scholar 

  14. Chen ZY, Fan ZP, Sun M (2012) A hierarchical multiple kernel support vector machine for customer churn prediction using longitudinal behavioral data. Eur J Oper Res 223:461–472

    Article  MATH  MathSciNet  Google Scholar 

  15. Rakotomamonjy A, Bach FR, Canu S et al (2008) SimpleMKL. J Mach Learn Res 9:2491–2521

    MATH  MathSciNet  Google Scholar 

  16. Chen ZY, Fan ZP (2012) Distributed customer behavior prediction using multiplex data: a collaborative MK-SVM approach. Knowl Based Syst 35:111–119

    Article  Google Scholar 

  17. Wang S, Huang Q, Jiang S et al (2012) \(\text{ S }^{3}\)MKL: Scalable semi-supervised multiple kernel learning for real-world image applications. IEEE Trans Multimed 14:1259–1274

    Article  Google Scholar 

  18. Szafranski M, Grandvalet Y, Rakotomamonjy A (2010) Composite kernel learning. Mach Learn 79:73–103

    Article  MathSciNet  Google Scholar 

  19. Wu M, Scholkopf B, Bakir G (2006) A direct method for building sparse kernel learning algorithms. J Mach Learn Res 7:603–624

    MATH  MathSciNet  Google Scholar 

  20. Subrahmanya N, Shin YC (2010) Sparse multiple kernel learning for signal processing applications. IEEE Trans Pattern Anal Mach Intell 32:788–798

    Article  Google Scholar 

  21. Hu M, Chen Y, Kwok JTY (2009) Building sparse multiple-kernel SVM classifiers. IEEE Trans Neural Netw 20:827–839

    Google Scholar 

  22. Chen ZY, Li JP, Wei LW et al (2011) Multiple kernel support vector machine based multiple tasks oriented data mining system for gene expression data analysis. Expert Syst Appl 38:12151–12159

    Article  Google Scholar 

  23. Gonen M, Alpaydın E (2010) Supervised learning of local projection kernels. Neurocomputing 73:1694–1703

    Article  Google Scholar 

  24. Domeniconi C, Peng J, Yan BJ (2011) Composite kernels for semi-supervised clustering. Knowl Inf Syst 28:99–116

    Article  Google Scholar 

  25. Zhu X, Li B, Wu X et al (2011) CLAP: collaborative pattern mining for distributed information systems. Decis Support Syst 52:40–51

    Article  Google Scholar 

  26. Marston S, Li Z, Bandyopadhyay S et al (2010) Cloud computing—the business perspective. Decis Support Syst 51:176–189

    Article  Google Scholar 

  27. Graf HP, Cosatto E, Bottou L et al (2005) Parallel support vector machines: the cascade SVM. Adva Neural Inf Process Syst 17:521–528

    Google Scholar 

  28. Alham NK, Li M, Liu Y et al (2012) A distributed SVM ensemble for image: classification and annotation. In: Proceedings of the 9th international conference on fuzzy systems and knowledge discovery. IEEE, Piscataway, NJ, USA, pp 1581–1584

  29. Chang EY, Zhu K, Wang H et al (2007) PSVM: parallelizing support vector machines on distributed computers. Adv Neural Inf Process Syst 20:1–8

    Google Scholar 

  30. Chu CT, Kim SK, Lin YA et al (2006) Map-reduce for machine learning on multicore. Adv Neural Inf Process Syst 17:281–288

    Google Scholar 

  31. Forero PA, Cano A, Giannakis GB (2010) Consensus-based distributed support vector machines. J Mach Learn Res 11:1663–1707

    MATH  MathSciNet  Google Scholar 

  32. Lu Y, Roychowdhury V, Vandenberghe L (2008) Distributed parallel support vector machines in strongly connected networks. IEEE Trans Neural Netw 19:1167–1178

    Google Scholar 

  33. Zanni L, Serafini T, Zanghirati G (2006) Parallel software for training large scale support vector machines on multiprocessor systems. J Mach Learn Res 7:1467–1492

    MATH  MathSciNet  Google Scholar 

  34. Woodsend K, Gondzio J (2009) Hybrid MPI/OpenMP parallel linear support vector machine training. J Mach Learn Res 10:1937–1953

    MATH  MathSciNet  Google Scholar 

  35. Bertsekas DP, Tsitsiklis JN (1989) Parallel and distributed computation: numerical methods. Prentice Hall, Upper Saddle River

    MATH  Google Scholar 

  36. Boyd S, Parikh N, Chu E et al (2010) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Mach Learn 3:1–122

    Article  MATH  Google Scholar 

  37. Nedic A, Ozdaglar A (2010) Cooperative distributed multi-agent optimization. In: Palomar DP, Eldar YC (eds) Convex optimization in signal processing and communications. Cambridge University Press, Cambridge

    Google Scholar 

  38. Frank A, Asuncion A (2010) UCI machine learning repository. http://archive.ics.uci.edu/ml

  39. Mota JFC, Xavier JMF, Aguiar PMQ et al (2012) D-ADMM: a communication-efficient distributed algorithm for separable optimization. http://arxiv.org/abs/1202.2805

  40. Martins AFT, Smith NA, Xing EP et al (2011) Online learning of structured predictors with multiple kernels. J Mach Learn Res 15:507–515

    Google Scholar 

  41. Suzuki T, Tomioka R (2011) SpicyMKL: a fast algorithm for multiple kernel learning with thousands of kernels. Mach Learn 85:77–108

    Article  MATH  MathSciNet  Google Scholar 

  42. Cao LJ, Keerthi SS, Ong CJ, Zhang JQ et al (2006) Parallel sequential minimal optimization for the training of support vector machines. IEEE Trans Neural Netw 17:1039–1049

    Google Scholar 

Download references

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China (Project No. 71101023, 71021061, 71271051) and the Fundamental Research Funds for the Central Universities, NEU, China (Project No. N120406001, N110706001). The author would like to express his sincere thanks to referees for their constructive comments and suggestions.

Author information

Authors and Affiliations

  1. Department of Management Science and Engineering, School of Business Administration, Northeastern University, Shenyang,  110819, China

    Zhen-Yu Chen & Zhi-Ping Fan

Authors
  1. Zhen-Yu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhi-Ping Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhen-Yu Chen.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chen, ZY., Fan, ZP. Parallel multiple kernel learning: a hybrid alternating direction method of multipliers. Knowl Inf Syst 40, 673–696 (2014). https://doi.org/10.1007/s10115-013-0655-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-013-0655-5

Keywords

Search

Navigation