Skip to main content
SpringerLink
Log in

Integrating Spectral Kernel Learning and Constraints in Semi-Supervised Classification

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Recently, integrating new knowledge sources such as pairwise constraints into various classification tasks with insufficient training data has been actively studied in machine learning. In this paper, we propose a novel semi-supervised classification approach, called semi-supervised classification with enhanced spectral kernel, which can simultaneously handle both sparse labeled data and additional pairwise constraints together with unlabeled data. Specifically, we first design a non-parameter spectral kernel learning model based on the squared loss function. Then we develop an efficient semi-supervised classification algorithm which takes advantage of Laplacian spectral regularization: semi-supervised classification with enhanced spectral kernel under the squared loss (ESKS). Finally, we conduct many experiments on a variety of synthetic and real-world data sets to demonstrate the effectiveness of the proposed ESKS algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Chapelle O, Schölkopf B, Zien A (2006) Semi-supervised learning. The MIT Press, Cambridge

    Google Scholar 

  2. Zhu X (2008) Semi-supervised learning literature survey. Computer Sciences Technical Report 1530, University of Wisconsin, Madison

  3. Zhu X, Ghahramani Z, Lafferty J (2003) Semi-supervised learning using Gaussian fields and harmonic functions. In: Proceedings of the 20th international conference on machine learning (ICML), pp 912–919

  4. Zhou D, Bousquet O, Lal T, Weston J, Schölkopf B (2004) Learning with local and global consistency. In: Advances in Neural Information Processing Systems (NIPS), pp 321–328

  5. Belkin M, Niyogi P, Sindhwani V (2006) Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J Mach Learn Res 7: 2399–2434

    MathSciNet  MATH  Google Scholar 

  6. Melacci S, Belkin M (2011) Laplacian support vector machines trained in the primal. J Mach Learn Res 12: 1149–1184

    MathSciNet  Google Scholar 

  7. Yan R, Zhang J, Yang J, Hauptmann A (2006) A discriminative learning framework with pairwise constraints for video object classification. IEEE Trans Pattern Anal Mach Intell 28(4): 578–593

    Article  Google Scholar 

  8. Nguyen N, Garuana R (2008) Improving classification with pairwise constraints: A margin-based approach. In: Proceedings of the European conference on machine learning and knowledge discovery in databases, pp 113–124

  9. Li Z, Liu J, Tang X (2008) Pairwise constraint propagation by semidefinite programming for semi-supervised classification. In: Proceedings of the 25th international conference on machine learning (ICML), pp 576–583

  10. Wagstaff K, Cardie C (2000) Clustering with instance-level constraints. In: Proceedings of the 17th internationl conference on machine learning (ICML), pp 1103–1110

  11. Klein D, Kamvar S, Manning C (2002) From instance-level constraints to space-level constraints: making the most of prior knowledge in data clustering. In: Proceedings of the 19th international conference on machine learning (ICML), pp 307–314

  12. Wang F, Li T, Zhang C (2008) Semi-supervised clustering via matrix factorization. In: Proceedings of the 8th SIAM international conference on data mining (SDM), pp 1–12

  13. Li Z, Liu J, Tang X (2009) Constrained clustering via spectral regularization. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 421–428

  14. Shang F, Liu Y, Wang F (2011) Learning spectral embedding for semisupervised clustering. In: Proceedings of the 11th IEEE international conference on data mining (ICDM), pp 597–606

  15. Xing E, Ng A, Jordan M, Russell S (2003) Distance metric learning, with application to clustering with side-information. In: Advances in Neural Information Processing Systems (NIPS), pp 505–512

  16. Wang F, Chen S, Zhang C, Li T (2008) Semi-supervised metric learning by maximizing constraint margin. In: Proceedings of the 17th ACM international conference on information and knowledge management (CIKM), pp 1457–1458

  17. Bian, W, Tao D (2011) Learning a distance metric by empirical loss minimization. In: Proceedings of the 22nd international joint conference on artificial intelligence (IJCAI), pp 1186–1191

  18. Bian W, Tao D (2011) Max-min distance analysis by using sequential SDP relaxation for dimension reduction. IEEE Trans Pattern Anal Mach Intell 33(5): 1037–1050

    Article  Google Scholar 

  19. Wang F, Sun J, Ebadollahi S (2011) Integrating distance metrics learned from multiple experts and its application in inter-patient similarity assessment. In: Proceedings of the 11th SIAM international conference on data mining (SDM), pp 59–70

  20. Jain P, Kulis B, Davis JV, Dhillon IS (2012) Metric and kernel learning using a linear transformation. J Mach Learn Res 13: 519–547

    Google Scholar 

  21. Zhuang J, Tsang I, Hoi SCH (2011) A family of simple non-parametric kernel learning algorithms. J Mach Learn Res 12: 1313–1347

    MathSciNet  Google Scholar 

  22. Zhu X, Kandola JS, Ghahramani Z, Lafferty JD (2005) Nonparametric transforms of graph kernels for semi-supervised learning. In: Advances in Neural Information Processing Systems (NIPS), pp 1641–1648

  23. Zhang T, Ando R (2005) Analysis of spectral kernel design based semi-supervised learning. In: Advances in Neural Information Processing Systems (NIPS), pp 1601–1608

  24. Hoi S, Jin R, Lyu M (2007) Learning nonparametric kernel matrices from pairwise constraints. In: Proceedings of the 24th international conference on machine learning (ICML), pp 361–368

  25. Liu W, Qian B, Cui J, Liu J (2009) Spectral kernel learning for semi-supervised classification. In: Proceedings of the 21st international joint conference on artificial intelligence (IJCAI), pp 1150–1155

  26. Hu E, Chen S, Zhang D, Yin X (2010) Semisupervised kernel matrix learning by kernel propagation. IEEE Trans Neural Netw 21(11): 1831–1841

    Article  Google Scholar 

  27. Shang F, Jiao LC, Shi J, Wang F, Gong M (2012) Fast affinity propagation clustering: a multilevel approach. Pattern Recognit 45(1): 474–486

    Article  Google Scholar 

  28. Shang F, Jiao LC, Shi J, Gong M, Shang R (2011) Fast density-weighted low-rank approximation spectral clustering. Data Min Knowl Discov 23(2): 345–378

    Article  MathSciNet  MATH  Google Scholar 

  29. Guan N, Tao D, Luo Z, Yuan B (2011) Manifold regularized discriminative nonnegative matrix factorization with fast gradient descent. IEEE Trans Image Process 20(7): 2030–2048

    Article  MathSciNet  Google Scholar 

  30. Guan N, Tao D, Luo Z, Yuan B (2011) Non-negative patch alignment framework. IEEE Trans Neural Netw 22(8): 1218–1230

    Article  Google Scholar 

  31. Shang F, Jiao LC, Wang F (2012) Graph dual regularization non-negative matrix factorization for co-clustering. Pattern Recognit 45(6): 2237–2250

    Article  MATH  Google Scholar 

  32. Weinberger KQ, Sha F, Zhu Q, Saul LK (2007) Graph Laplacian regularization for large-scale semidefinite programming. In: Advances in Neural Information Processing Systems (NIPS), pp 1489–1496

  33. Wu X-M, So A, Li Z, Li S (2009) Fast graph Laplacian regularized kernel learning via semidefinite-quadratic-linear programming. In: Advances in Neural Information Processing Systems (NIPS), pp 1964–1972

  34. Golub GH, Loan CFV (1996) Matrix computations, 3rd edn. Johns Hopkins University Press, Baltimore

    MATH  Google Scholar 

  35. Tütüncü RH, Toh KC, Todd MJ (2003) Solving semidefinite-quadratic-linear programs using SDPT3. Math Program 95: 189–217

    Article  MathSciNet  MATH  Google Scholar 

  36. Chang C, Lin C (2001) LIBSVM: A library for support vector machines. Available at http://www.csie.ntu.edu.tw/~cjlin/libsvm

  37. Zelnik-Manor L, Perona P (2004) Self-tuning spectral clustering. In: Advances in Neural Information Processing Systems (NIPS), pp 1601–1608

  38. Souvenir R, Pless R (2005) Manifold clustering. In: Proceedings of the 10th international conference on computer vision (ICCV), pp 648–654

  39. LeCun Y, Cortes C (2009) The MNIST database of handwritten digits. Available: http://yann.lecun.com/exdb/mnist/

  40. Nene SA, Nayar SK, Murase J (1996) Columbia object image library (COIL-20). Technical Report CUCS-005-96, Columbia University

  41. Vidal R, Ma Y, Piazzi J (2004) A new GPCA algorithm for clustering subspaces by fitting, differentiating and dividing polynomials. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 510–517

  42. Fei-Fei L, Fergus R, Perona P (2004) Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  43. Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: spatial pyramid matching for recognition natural scene categories. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2169–2178

  44. Georghiades AS, Belhumeur PN, Kriegman DJ (2001) From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Trans Pattern Anal Mach Intell 23(6): 643–660

    Article  Google Scholar 

  45. Szummer M, Jaakkola T (2002) Partially labeled classification with Markov random walks. In: Advances in Neural Information Processing Systems (NIPS), pp 945–952

Download references

Author information

Authors and Affiliations

  1. Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education of China, Xidian University, Mailbox 224, No. 2 South TaiBai Road, Xi’an, 710071, China

    Fanhua Shang, L. C. Jiao & Yuanyuan Liu

Authors
  1. Fanhua Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. L. C. Jiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuanyuan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fanhua Shang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Shang, F., Jiao, L.C. & Liu, Y. Integrating Spectral Kernel Learning and Constraints in Semi-Supervised Classification. Neural Process Lett 36, 101–115 (2012). https://doi.org/10.1007/s11063-012-9224-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-012-9224-2

Keywords

Search

Navigation