Abstract
Graph-based methods have aroused wide interest in pattern recognition and machine learning, which capture the structural information in data into classifier design through defining a graph over the data and assuming label smoothness over the graph. Laplacian Support Vector Machine (LapSVM) is a representative of these methods and an extension of the traditional SVM by optimizing a new objective additionally appended Laplacian regularizer. The regularizer utilizes the local linear patches to approximate the data manifold structure and assumes the same label of the data on each patch. Though LapSVM has shown more effective classification performance than SVM experimentally, it in fact concerns more the locality than the globality of data manifold due to the Laplacian regularizer itself. As a result, LapSVM is relatively sensitive to the local change of the data and cannot characterize the manifold quite faithfully. In this paper, we design an alternative regularizer, termed as Glocalization Pursuit Regularizer. The new regularizer introduces a natural global structure measure to grasp the global and local manifold information as simultaneously as possible, which can be proved to make the representation of the manifold more compact than the Laplacian regularizer. We further introduce the new regularizer into SVM to develop an alternative graph-based SVM, called as Glocalization Pursuit Support Vector Machine (GPSVM). GPSVM not only inherits the advantages of both SVM and LapSVM but also uses the structural information more reasonably to guide the classifier design. The experiments both on the toy and real-world datasets demonstrate the better classification performance of our proposed GPSVM compared with SVM and LapSVM.
Similar content being viewed by others
References
Zhu X (2008) Semi-supervised learning literature survey. Technical report, 1530, Madison: Department of Computer Sciences, University of Wisconsin
Bousquet O, Chapelle O, Hein M (2003) Measure based regularization. NIPS, Canada
Xue H, Chen S, Zeng X (2008) Classifier learning with a new locality regularization method. Pattern Recogn 41(5):1496–1507
Xue H, Chen S, Yang Q (2009) Discriminatively regularized least-squares classification. Pattern Recogn 42(1):93–104
Wu M, Schölkopf B (2007) Transductive classification via local learning regularization. Eleventh international conference on artificial intelligence and statistics (AISTATS)
Haykin S (2001) Neural networks: a comprehensive foundation. Tsinghua University Press, Beijing
Zhang T (2004) Statistical behavior and consistency of classification methods based on convex risk minimization. Ann Stat 32:56–85
Zhang T (2004) Statistical analysis of some multicategory large margin classification methods. J Mach Learn Res 5:1225–1251
Rosasco L, Vito ED, Caponnetto A, Piana M, Verri A (2004) Are loss functions all the same? Neural Comput 16(5):1063–1076
Rosset S, Zhu J, Hastie T (2003) Margin maximizing loss functions. NIPS, Canada
Chen Z, Haykin S (2002) On different facets of regularization theory. Neural Comput 14(12):2791–2846
Belkin M, Niyogi P Laplacian (2001) Eigenmaps and spectral technique for embedding and clustering. NIPS, 15: Vancouver, British Columbia, Canada
Belkin M, Niyogi P, Sindhwani V (2005) On mainfold regularization. In: Proceedings of the 10th international workshop on artificial intelligence and statistics (AISTATS), Savannah Hotel, Barbados, 17–24
Zhou D, Bousquet O, Lal TN, Weston J, Schölkopf B (2004) Learning with local and global consistency. In: Thrun S, Saul L, Scholkopf B (eds) Advances in neural information processing systems 16. MIT Press, Cambridge, pp 321–328
Holder LB, Cook DJ (2003) Graph-based relational learning: current and future directions. ACM SIGKDD Explor Newsl 5(1):90–93
Cook DJ, Holder LB (2000) Graph-based data mining. IEEE Intell Syst 15:32–41
Zhu X (2005) Semi-supervised learning with graphs. PhD Thesis, Carnegie Mellon University. CMU-LTI-05-192
Belkin M, Matveeva I, P. Niyogi (2004) Regularization and semi-supervised learning on large graphs. COLT
Vapnik V (1998) Statistical learning theory. Wiley, New York
Cristianini N, Shawe-Taylor J (2004) An introduction to support vector machines and other kernel-based learning methods. Publishing House of Electronics Industry, Beijing
Zhu X, Ghahramani Z, Lafferty J (2003) Semi-supervised learning using Gaussian fields and harmonic functions. In: Proceedings of the 20th international conference on machine learning (ICML’03), Washington, DC, 912–919
Zhu X, Ghahramani Z (2002) Towards semi-supervised classification with Markov random fields. Technical report CMU-CALD-02-106. Carnegie Mellon University
Belkin M, Niyogi P, Sindhwani V (2004) Manifold regularization: A geometric framework for learning from examples. Department of Computer Science, University of Chicago, Tech. Rep: TR-2004-06
Xue H, Chen S (2007) Alternative robust local embedding. In: The international conference on wavelet analysis and pattern recognition (ICWAPR), 591–596
Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(22):2323–2326
Schölkopf B, Smola AJ (2002) Learning with Kernels. 644. MIT Press, Cambridge
Belkin M (2003) Problems of learning on manifolds. PhD Thesis, The University of Chicago
Cristianini N, Shawe-Taylor J (2000) An introduction to support vector machines and other kernel-based learning methods. Cambridge University Press, Cambridge
Xue H, Chen S, Yang Q (2008) Structural support vector machine. In: The 15th international symposium on neural networks (ISNN), Part I, LNCS, vol 5263, pp 501–511
Acknowledgments
This work was supported by National Natural Science Foundations of China (Grant Nos. 60773061, 60905002, 60973097 and 61035003) and Natural Science Foundations of Jiangsu Province of China (Grant No. BK2008381).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Xue, H., Chen, S. Glocalization pursuit support vector machine. Neural Comput & Applic 20, 1043–1053 (2011). https://doi.org/10.1007/s00521-010-0448-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-010-0448-7