Skip to main content
SpringerLink
Log in

Laplacian multiset canonical correlations for multiview feature extraction and image recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Multiset canonical correlation analysis (MCCA) aims at revealing the linear correlations among multiple sets of high-dimensional data. Therefore, it is only a linear multiview dimensionality reduction technique and such a linear model is insufficient to discover the nonlinear correlation information hidden in multiview data. In this paper, we incorporate the local structure information into MCCA and propose a novel algorithm for multiview dimensionality reduction, called Laplacian multiset canonical correlations (LapMCCs), which simultaneously considers local within-view and local between-view correlations by using nearest neighbor graphs. This makes LapMCC capable of discovering the nonlinear correlation information among multiview data by combining many locally linear problems together. Moreover, we also develop an orthogonal version of LapMCC to preserve the metric structure. The proposed LapMCC method is applied to face and object image recognition. The experimental results on AR, Yale-B, AT&T, and ETH-80 databases demonstrate the superior performance of LapMCC compared to existing multiview dimensionality reduction methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. In SemiCCA, the real meaning of “semi-supervised” is actually “semi-paired”. That is, a part of two view data from the same objects are paired, and the other are unpaired.

  2. To recursively solve all sets of projection directions, except the first set of ones, Kettenring [32] gave three classes of orthogonal constraints:

    1. (a).

      α T it S ii α i  = 0, i = 1, 2, ⋯, m; t = 1, 2, ⋯, k − 1;

    2. (b).

      α T it S ii α i  = 0, \( i\in {\mathcal{A}}_q \); t = 1, 2, ⋯, k − 1, where \( {\mathcal{A}}_q \) is a nonempty subset of q of the first m integers;

    3. (c).

      α T jt S ji α i  = 0, i, j = 1, 2, ⋯, m; t = 1, 2, ⋯, k − 1.

    In this paper, we choose the constraint (a) to compute higher-stage projection directions, as used in [22, 32].

  3. Matlab code available at http://www.ms.k.u-tokyo.ac.jp/software.html

  4. Matlab code available at http://vipl.ict.ac.cn/resources/codes

  5. http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html

References

  1. Bach FR, Jordan MI (2002) Kernel independent component analysis. Journal of Machine Learning Research 3:1–48

    MathSciNet  MATH  Google Scholar 

  2. Belkin M, Niyogi P (2001) Laplacian eigenmaps and spectral techniques for embedding and clustering. In: Advances in Neural Information Processing Systems. MIT Press, Cambridge, pp 585–591

    Google Scholar 

  3. Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation 15(6):1373–1396

    Article  MATH  Google Scholar 

  4. Chaudhuri K, Kakade SM, Livescu K, Sridharan K (2009) Multi-view clustering via canonical correlation analysis. In: Proceedings of the 26th annual international conference on machine learning. ACM, New York, pp 129–136

  5. Chu D, Liao L-Z, Ng MK, Zhang X (2013) Sparse canonical correlation analysis: new formulation and algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(12):3050–3065

    Article  Google Scholar 

  6. Chu MT, Watterson JL (1993) On a multivariate eigenvalue problem: I. algebraic theory and power method. SIAM Journal on Scientific Computing 14(5):1089–1106

    Article  MathSciNet  MATH  Google Scholar 

  7. Chung FRK (1997) Spectral graph theory. Am. Math. Soc, RI

    MATH  Google Scholar 

  8. Donner R, Reiter M, Langs G, Peloschek P, Bischof H (2006) Fast active appearance model search using canonical correlation analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(10):1690–1694

    Article  Google Scholar 

  9. Donoho DL, Grimes C (2003) Hessian eigenmaps: new locally linear embedding techniques for high-dimensional data. Proc Nat Acad Sci USA 100(10):5591–5596

    Article  MATH  Google Scholar 

  10. Duchene J, Leclercq S (1988) An optimal transformation for discriminant and principal component analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 10(6):978–983

    Article  MATH  Google Scholar 

  11. Foster DP, Kakade SM, Zhang T (2008) Multi-view dimensionality reduction via canonical correlation analysis. Technical Report TTI-TR-2008-4. Toyota Technological Institute at Chicago (TTIC), USA

    Google Scholar 

  12. Friman O, Cedefamn J, Lundberg P, Borga M, Knutsson H (2001) Detection of neural activity in functional MRI using canonical correlation analysis. Magnetic Resonance in Medicine 45(2):323–330

    Article  Google Scholar 

  13. Fu Y, Yan S, Huang TS (2008) Correlation metric for generalized feature extraction. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(12):2229–2235

    Article  Google Scholar 

  14. Fukumizu K, Bach FR, Gretton A (2007) Statistical consistency of kernel canonical correlation analysis. Journal of Machine Learning Research 8:361–383

    MathSciNet  MATH  Google Scholar 

  15. Georghiades A, Belhumeur P, Kriegman D (2001) From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(6):643–660

    Article  Google Scholar 

  16. Gui J, Tao D, Sun Z, Luo Y, You X, Tang YY (2014) Group sparse multiview patch alignment framework with view consistency for image classification. IEEE Transactions on Image Processing 23(7):3126–3137

    Article  MathSciNet  Google Scholar 

  17. Hardoon DR, Shawe-Tayler J (2011) Sparse canonical correlation analysis. Machine Learning 83(3):331–353

    Article  MathSciNet  MATH  Google Scholar 

  18. Hardoon DR, Shawe-Taylor J (2009) Convergence analysis of kernel canonical correlation analysis: theory and practice. Machine Learning 74(1):23–38

    Article  Google Scholar 

  19. Hardoon DR, Szedmak S, Shawe-Taylor J (2004) Canonical correlation analysis: an overview with application to learning methods. Neural Computation 16(12):2639–2664

    Article  MATH  Google Scholar 

  20. He X, Niyogi P (2003) Locality preserving projections. In: Advances in neural information processing systems. MIT Press, Cambridge

    Google Scholar 

  21. He X, Yan S, Hu Y, Niyogi P, Zhang HJ (2005) Face recognition using laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(3):328–340

    Article  Google Scholar 

  22. Horst P (1961) Relations among m sets of measures. Psychometrika 26(2):129–149

    Article  MathSciNet  MATH  Google Scholar 

  23. Hotelling H (1936) Relations between two sets of variates. Biometrika 28(34):321–377

    Article  MATH  Google Scholar 

  24. Hou S, Sun Q, Xia D (2011) Feature fusion using multiple component analysis. Neural Processing Letters 34(3):259–275

    Article  Google Scholar 

  25. Hou C, Zhang C, Wu Y, Nie F (2010) Multiple view semi-supervised dimensionality reduction. Pattern Recognition 43(3):720–730

    Article  MATH  Google Scholar 

  26. Huang H, He H, Fan X, Zhang J (2010) Super-resolution of human face image using canonical correlation analysis. Pattern Recognition 43(7):2532–2543

    Article  MATH  Google Scholar 

  27. Jain AK, Duin RPW, Mao J (2000) Statistical pattern recognition: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(1):4–37

    Article  Google Scholar 

  28. Jing X, Li S, Lan C, Zhang D, Yang J, Liu Q (2011) Color image canonical correlation analysis for face feature extraction and recognition. Signal Processing 91:2132–2140

    Article  MATH  Google Scholar 

  29. Jolliffe IT (2002) Principal component analysis, 2nd edn. Springer, New York

    MATH  Google Scholar 

  30. Kakade SM, Foster DP (2007) Multi-view regression via canonical correlation analysis. In: Proceedings of the 20th annual conference on learning theory. Springer, Berlin, pp 82–96

  31. Kan M, Shan S, Zhang H, Lao S, Chen X (2015) Multi-view discriminant analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence. doi:10.1109/TPAMI.2015.2435740

    Google Scholar 

  32. Kettenring JR (1971) Canonical analysis of several sets of variables. Biometrika 58(3):433–451

    Article  MathSciNet  MATH  Google Scholar 

  33. Kim TK, Cipolla R (2009) Canonical correlation analysis of video volume tensors for action categorization and detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(8):1415–1428

    Article  Google Scholar 

  34. Kim TK, Kittler J, Cippola R (2007) Discriminative learning and recognition of image set classes using canonical correlations. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6):1005–1018

    Article  Google Scholar 

  35. Kimura A, Kameoka H, Sugiyama M, Nakano T, Maeda E, Sakano H, Ishiguro K (2010) SemiCCA: Efficient semi-supervised learning of canonical correlations. In: Proceedings of the 20th international conference on pattern recognition. IEEE, Istanbul, pp 2933–2936

  36. Lai PL, Fyfe C (2000) Kernel and nonlinear canonical correlation analysis. International Journal of Neural Systems 10(5):365–377

    Article  Google Scholar 

  37. Lee K, Ho J, Kriegman D (2005) Acquiring linear subspaces for face recognition under variable lighting. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(5):684–698

    Article  Google Scholar 

  38. Leibe B, Schiele B (2003) Analyzing appearance and contour based methods for object categorization. In: Proceedings of IEEE conference on computer vision and pattern recognition. IEEE, Wisconsin, pp 409–415

  39. Liu W, Li Y, Lin X, Tao D, Wang Y (2014) Hessian-regularized co-training for social activity recognition. PLoS ONE 9(9), e108474

    Article  Google Scholar 

  40. Liu W, Liu H, Tao D, Wang Y, Lu K (2015) Multiview Hessian regularized logistic regression for action recognition. Signal Processing 110:101–107

    Article  Google Scholar 

  41. Liu W, Tao D (2013) Multiview hessian regularization for image annotation. IEEE Transactions on Image Processing 22(7):2676–2687

    Article  MathSciNet  Google Scholar 

  42. Liu W, Tao D, Cheng J, Tang Y (2014) Multiview Hessian discriminative sparse coding for image annotation. Computer Vision and Image Understanding 118:50–60

    Article  Google Scholar 

  43. Long B, Yu PS, Zhang Z (2008) A general model for multiple view unsupervised learning. In: Proceedings of the 2008 SIAM international conference on data mining (SDM). SIAM, Newport Beach, pp 822–833

  44. Martinez AM, Benavente R (1998) The AR face database. Technical Report #24. Computer Vision Center, Autonomous Univ, Barcelona

    Google Scholar 

  45. Martinez AM, Kak AC (2001) PCA versus LDA. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(2):228–233

    Article  Google Scholar 

  46. Melzer T, Reiter M, Bischof M (2003) Appearance models based on kernel canonical correlation analysis. Pattern Recognition 36(9):1961–1971

    Article  MATH  Google Scholar 

  47. Nielsen AA (2002) Multiset canonical correlations analysis and multispectral, truly multitemporal remote sensing data. IEEE Transactions on Image Processing 11(3):293–305

    Article  MathSciNet  Google Scholar 

  48. Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326

    Article  Google Scholar 

  49. Rupnik J, Shawe-Taylor J (2010) Multi-view canonical correlation analysis. In: SiKDD. Available at http://ailab.ijs.si/dunja/SiKDD2010/Papers/Rupnik_Final.pdf

  50. Sargin ME, Yemez Y, Erzin E (2007) Audiovisual synchronization and fusion using canonical correlation analysis. IEEE Transactions on Multimedia 9(7):1396–1403

    Article  Google Scholar 

  51. Schölkopf B, Smola A, Müller KR (1998) Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation 10(5):1299–1319

    Article  Google Scholar 

  52. Sharma A, Kumar A, Daume H, Jacobs DW (2012) Generalized multiview analysis: a discriminative latent space. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Providence, pp 2160–2167.

  53. Sugiyama M (2006) Local Fisher discriminant analysis for supervised dimensionality reduction. In: Proceedings of the 23rd International Conference on Machine Learning. ACM, New York, pp 905–912

  54. Sugiyama M (2007) Dimensionality reduction of multimodal labeled data by local Fisher discriminant analysis. Journal of Machine Learning Research 8:1027–1061

    MATH  Google Scholar 

  55. Sugiyama M, Idé T, Nakajima S, Sese J (2010) Semi-supervised local Fisher discriminant analysis for dimensionality reduction. Machine Learning 78:35–61

    Article  MathSciNet  Google Scholar 

  56. Sun TK, Chen SC (2007) Locality preserving CCA with applications to data visualization and pose estimation. Image and Vision Computing 25(5):531–543

    Article  Google Scholar 

  57. Sun T, Chen S, Yang J, Shi P (2008) A novel method of combined feature extraction for recognition. In: Proceedings of the 8th IEEE conference on data mining. IEEE, Pisa, pp 1043–1048

  58. Sun L, Ji S, Ye J (2010) Canonical correlation analysis for multilabel classification: a least-squares formulation, extensions, and analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(1):194–200

    Google Scholar 

  59. Sun Q-S, Zeng S-G, Liu Y, Heng P-A, Xia D-S (2005) A new method of feature fusion and its application in image recognition. Pattern Recognition 38(12):2437–2448

    Article  Google Scholar 

  60. Tao D, Jin L, Liu W, Li X (2013) Hessian regularized support vector machines for mobile image annotation on the cloud. IEEE Transactions on Multimedia 15(4):833–844

    Article  Google Scholar 

  61. Tenenbaum JB, Silva V, Langford JC (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290(5500):2319–2323

    Article  Google Scholar 

  62. Worsley KJ, Poline JB, Friston KJ, Evans AC (1997) Characterizing the response of PET and fMRI data using multivariate linear models. NeuroImage 6(4):305–319

    Article  Google Scholar 

  63. Xia T, Tao D, Mei T, Zhang Y (2010) Multiview spectral embedding. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 40(6):1438–1446

    Article  Google Scholar 

  64. Xie B, Mu Y, Tao D, Huang K (2011) m-SNE: multiview stochastic neighbor embedding. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 41(4):1088–1096

    Article  Google Scholar 

  65. Xu C, Tao D, Xu C (2014) Large-margin multi-view information bottleneck. IEEE Transactions on Pattern Analysis and Machine Intelligence 36(8):1559–1572

    Article  Google Scholar 

  66. Xu C, Tao D, Xu C (2015) Multi-view intact space learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. doi:10.1109/TPAMI.2015.2417578

    Google Scholar 

  67. Yan S, Xu D, Zhang B, Zhang H-J, Yang Q, Lin S (2007) Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(1):40–51

    Article  Google Scholar 

  68. Yang J, Zhang D, Yang JY, Niu B (2007) Globally maximizing, locally minimizing: unsupervised discriminant projection with applications to face and palm biometrics. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(4):650–664

    Article  Google Scholar 

  69. Yuan Y-H, Sun Q-S (2012) Discriminative learning of multiset integrated canonical correlation analysis for feature fusion. In: Proceedings of the 15th International Conference on Information Fusion (ICIF). IEEE, Piscataway, pp 882–887

  70. Yuan Y-H, Sun Q-S (2014) Multiset canonical correlations using globality preserving projections with applications to feature extraction and recognition. IEEE Transactions on Neural Networks and Learning Systems 25(6):1131–1146

    Article  Google Scholar 

  71. Yuan Y-H, Sun Q-S, Zhou Q, Xia D-S (2011) A novel multiset integrated canonical correlation analysis framework and its application in feature fusion. Pattern Recognition 44(5):1031–1040

    Article  MATH  Google Scholar 

  72. Yuan Y-H, Sun Q-S, Ge H-W (2014) Fractional-order embedding canonical correlation analysis and its applications to multi-view dimensionality reduction and recognition. Pattern Recognition 47(3):1411–1424

    Article  MATH  Google Scholar 

  73. Zhang LH, Chu MT (2009) On a multivariate eigenvalue problem: II. Global solutions and the Gauss-Seidel method. Available at http://www4.ncsu.edu/~mtchu/Research/Papers/Readme.html

  74. Zhu X, Huang Z, Shen HT, Cheng J, Xu C (2012) Dimensionality reduction by mixed kernel canonical correlation analysis. Pattern Recognition 45(8):3003–3016

    Article  MATH  Google Scholar 

Download references

Acknowledgments

This work is supported by the National Natural Science Foundation of China under Grant Nos. 61402203, 61273251, and 61305017, the Fundamental Research Funds for the Central Universities under Grant No. JUSRP11458, and the Program for New Century Excellent Talents in University under Grant No. NCET-12-0881. Moreover, we would like to thank the editor and all of the anonymous reviewers for their constructive comments, which significantly improved the quality of this paper.

Author information

Authors and Affiliations

  1. School of Information Engineering, Yangzhou University, Yangzhou, 225127, China

    Yun-Hao Yuan & Yun Li

  2. Department of Computer Science and Technology, Jiangnan University, Wuxi, 214122, China

    Yun-Hao Yuan & Jin-Long Yang

  3. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China

    Xiao-Bo Shen & Quan-Sen Sun

  4. School of Information Technology and Electrical Engineering, The University of Queensland, Brisbane, QLD, 4072, Australia

    Xiao-Bo Shen

Authors
  1. Yun-Hao Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiao-Bo Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Quan-Sen Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jin-Long Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yun-Hao Yuan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yuan, YH., Li, Y., Shen, XB. et al. Laplacian multiset canonical correlations for multiview feature extraction and image recognition. Multimed Tools Appl 76, 731–755 (2017). https://doi.org/10.1007/s11042-015-3070-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-3070-y

Keywords

Search

Navigation