Skip to main content
SpringerLink
Log in

Transfer dimensionality reduction by Gaussian process in parallel

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Dimensionality reduction has been considered as one of the most significant tools for data analysis. In general, supervised information is helpful for dimensionality reduction. However, in typical real applications, supervised information in multiple source tasks may be available, while the data of the target task are unlabeled. An interesting problem of how to guide the dimensionality reduction for the unlabeled target data by exploiting useful knowledge, such as label information, from multiple source tasks arises in such a scenario. In this paper, we propose a new method for dimensionality reduction in the transfer learning setting. Unlike traditional paradigms where the useful knowledge from multiple source tasks is transferred through distance metric, we attempt to learn a more informative mapping function between the original data and the reduced data by Gaussian process that behaves more appropriately than other parametric regression methods due to its less parametric characteristic. In our proposal, we firstly convert the dimensionality reduction problem into integral regression problems in parallel. Gaussian process is then employed to learn the underlying relationship between the original data and the reduced data. Such a relationship can be appropriately transferred to the target task by exploiting the prediction ability of the Gaussian process model and inventing different kinds of regularizers. Extensive experiments on both synthetic and real data sets show the effectiveness of our method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. Actually, in the implementation, we are only interested in the learned kernel hyperparameters \(\theta ^i_l\)s.

  2. http://www.stanford.edu/~boyd/software.html.

  3. http://www.gaussianprocess.org/gpml/code/matlab/doc/index.html.

  4. http://archive.ics.uci.edu/ml/datasets/Wine+Quality/.

  5. http://people.csail.mit.edu/jrennie/20Newsgroups/.

References

  1. Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans PAMI 19:711–720

    Article  Google Scholar 

  2. Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15:1373–1396

    Article  MATH  Google Scholar 

  3. Bonilla EV, Chai KMA, Williams CKI (2008) Multi-task Gaussian process prediction. In: NIPS 20, pp 153–160

  4. Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  5. Cai D, He X, Han J (2007) Semi-supervised discriminant analysis. In: ICCV

  6. Cai D, He X, Han J (2007) Spectral regression: a unified approach for sparse subspace learning. In: ICDM, pp 73–82

  7. Chen B, Lam W, Tsang I, Wong T (2009) Extracting discriminative concepts for domain adaptation in text mining. In: KDD, pp 179–188

  8. Duan L, Tsang IW, Xu D, Chua T (2009) Domain adaptation from multiple sources via auxiliary classifiers. In: ICML

  9. Li T, Zhang C, Wang F, Chen S (2008) Semi-supervised metric learning by maximizing constraint margin. In: CIKM, pp 1457–1458

  10. Jurie F, Cevikalp H, Verbeek J, Klaser A (2008) Semi-supervised dimensionality reduction using pairwise equivalence constraints. In: VISAPP, pp 489–496

  11. He X, Cai D, Yan S, Zhang H-J (2005) Neighborhood preserving embedding. In: ICCV, pp 1208–1213

  12. He X, Yan S, Hu Y, Niyogi P, Zhang H-J (2005) Face recognition using laplacianfaces. IEEE Trans Pattern Anal Mach Intell 27(3):328–340

    Article  Google Scholar 

  13. Jin R, Wang S, Zhou Y (2009) Regularized distance metric learning: theory and algorithm. In: NIPS, pp 862–870

  14. Maaten LJP, Postma EO, Herick HJ (2009) Dimensionality reduction: a comparative review. Technical report TiCC-TR 2009–005, Tilburg University

  15. Pan SJ, Kwok JT, Yang Q (2008) Transfer learning via dimensionality reduction. In: AAAI, pp 677–682

  16. Pan SJ, Tsang IW, Kwok JT, Yang Q (2009) Domain adaptation via transfer component analysis. In: IJCAI, pp 1187–1192

  17. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans TKDE 22:1345–1359

    Google Scholar 

  18. Rasmussen CE, Williams CKI (2006) Gaussian processes for machine learning. MIT Press, Cambridge

    MATH  Google Scholar 

  19. Rish I, Grabarnik G, Cecchi G, Pereira F, Gordon GJ (2008) Closed-form supervised dimensionality reduction with generalized linear models. In: ICML, pp 832–839

  20. Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290:2323–2326

    Article  Google Scholar 

  21. Sajama, Orlitsky A (2005) Supervised dimensionality reduction using mixture models. In: ICML, pp 768–775

  22. Samaria FS, Harter AC (1994) Parameterisation of a stochastic model for human face identification. In: Applications of computer vision, 1994, Proceedings of the second IEEE workshop on, pp 138–142

  23. Slonim N, Tishby N (2000) Document clustering using word clusters via the information bottleneck method. In: SIGIR, pp 208–215

  24. Sugiyama M, Idé T, Nakajima S, Sese J (January 2010) Semi-supervised local fisher discriminant analysis for dimensionality reduction. J Mach Learn 78:35–61

    Google Scholar 

  25. Tenenbaum JB, de Silva V, Langford JC (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290:2319–2323

    Article  Google Scholar 

  26. Tong B, Shao H, Chou B-H, Suzuki E (2010) Semi-supervised projection clustering with transferred centroid regularization. In: ECML/PKDD (3), pp 306–321

  27. Tong B, Suzuki E (2010) Subclass-oriented dimension reduction with constraint transformation and manifold regularization. In: PAKDD (2), pp 1–13

  28. Wang Z, Song Y, Zhang C (2008) Transferred dimensionality reduction. In: ECML/PKDD, pp 550–565

  29. Yang X, Fu H, Zha H, Barlow J (2006) Semi-supervised nonlinear dimensionality reduction. In: ICML, pp 1065–1072

  30. Zha Z, Mei T, Wang M, Wang Z, Hua XS (2009) Robust distance metric learning with auxiliary knowledge. In: IJCAI, pp 1327–1332

  31. Zhang D, Zhou ZH, Chen S (2007) Semi-supervised dimensionality reduction. In: SDM, pp 629–634

  32. Zhang Y, Yeung D-Y (2010) A convex formulation for learning task relationships in multi-task learning. In: UAI, pp 733–742

  33. Zhang Y, Yeung D-Y (2010) Transfer metric learning by learning task relationships. In: KDD, pp 1199–1208

  34. Zhu X (2005) Semi-supervised learning literature survey. Technical report 1530, Computer Sciences, University of Wisconsin-Madison

  35. Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J R Stat Soc Ser B 67(2):301–320

    Article  MATH  MathSciNet  Google Scholar 

  36. Gower JC, Dijksterhuis GB (2004) Procrustes problem. Oxford University Press, Oxford

    Book  Google Scholar 

  37. Chen Y, Manjeet R, Ming D, Jing H (2008) Non-negative matrix factorization for semi-supervised data clustering. Knowl Inf Syst 17(3):355–379

    Article  Google Scholar 

  38. Keogh E, Chakrabarti K, Pazzani M, Mehrotra S (2000) Dimensionality reduction for fast similarity search in large time series databases. Knowl Inf Syst 3(3):263–286

    Article  Google Scholar 

  39. Song G, Cui B, Zheng B, Xie K, Yang D (2008) Accelerating sequence searching: dimensionality reduction method. Knowl Inf Syst 20(3):301–322

    Article  Google Scholar 

  40. Tong B, Gao J, Thach NH, Suzuki E (2011) Gaussian process for dimensionality reduction in transfer learning In: SDM, pp 783–794

  41. Zhou Z-H, Li M (2010) Semi-supervised learning by disagreement. Knowl Inf Syst 24(3):415–439

    Article  Google Scholar 

  42. Zhao W, He Q, Ma H, Shi Z (2011) Effective semi-supervised document clustering via active learning with instance-level constraints. Knowl Inf Syst 30(3):569–587

    Article  Google Scholar 

  43. Yan F, Qi Y (2010) Sparse Gaussian process regression via L1 penalization. In: ICML, pp 1183–1190

  44. Si S, Gao J, Tao D, Geng B (2010) Bregman divergence-based regularization for transfer subspace learning. IEEE Trans Knowl Data Eng 22(7):929–942

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graduate School of Systems Life Sciences, Kyushu University, Fukuoka, Japan

    Bin Tong, Thach Nguyen Huy & Hao Shao

  2. School of Computing and Mathematics, Charles Sturt University, Bathurst, Australia

    Junbin Gao

  3. Department of Informatics, ISEE, Kyushu University, Fukuoka, Japan

    Einoshin Suzuki

Authors
  1. Bin Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junbin Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thach Nguyen Huy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hao Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Einoshin Suzuki
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bin Tong.

Additional information

This work is supported by the grant-in-aid for scientific research on fundamental research (B) 21300053 from the Japanese Ministry of Education, Culture, Sports, Science and Technology, and Charles Sturt University Competitive Research Grant OPA 4818. This work was partially supported by the National Science Foundation (Grant No. 61133016), and the National High Technology Joint Research Program of China (863 Program, Grant No. 2011AA010706).

Rights and permissions

Reprints and permissions

About this article

Cite this article

Tong, B., Gao, J., Nguyen Huy, T. et al. Transfer dimensionality reduction by Gaussian process in parallel. Knowl Inf Syst 38, 567–597 (2014). https://doi.org/10.1007/s10115-012-0601-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-012-0601-y

Keywords

Search

Navigation