Skip to main content
SpringerLink
Log in

Isometric sliced inverse regression for nonlinear manifold learning

  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

Sliced inverse regression (SIR) was developed to find effective linear dimension-reduction directions for exploring the intrinsic structure of the high-dimensional data. In this study, we present isometric SIR for nonlinear dimension reduction, which is a hybrid of the SIR method using the geodesic distance approximation. First, the proposed method computes the isometric distance between data points; the resulting distance matrix is then sliced according to K-means clustering results, and the classical SIR algorithm is applied. We show that the isometric SIR (ISOSIR) can reveal the geometric structure of a nonlinear manifold dataset (e.g., the Swiss roll). We report and discuss this novel method in comparison to several existing dimension-reduction techniques for data visualization and classification problems. The results show that ISOSIR is a promising nonlinear feature extractor for classification applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Aizerman, M., Braverman, E., Rozonoer, L.: Theoretical foundations of the potential function method in pattern recognition learning. Autom. Remote Control 25, 821–837 (1964)

    MathSciNet  Google Scholar 

  • Alizadeh, A.A., et al.: Distinct types of diffuse large B-cell lymphoma identified by gene expression profiling. Nature 403(6769), 503–511 (2000)

    Article  Google Scholar 

  • Alon, U., et al.: Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc. Natl. Acad. Sci. USA 96(12), 6745–6750 (1999)

    Article  Google Scholar 

  • Balasubramanian, M., Schwartz, E.L.: The isomap algorithm and topological stability. Science 295(5552), 7 (2002)

    Article  Google Scholar 

  • Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15(6), 1373–1396 (2003)

    Article  MATH  Google Scholar 

  • Bengio, Y., Paiement, J., Vincent, P., Delalleau, O., Roux, N.L., Ouimet, M.: Out-of-sample extensions for LLE, isomap, MDS, eigenmaps, and spectral clustering. In: Neural Information Processing Systems, pp. 177–184. MIT Press, Cambridge (2003)

    Google Scholar 

  • Bian, W., Tao, D.: Manifold regularization for SIR with rate root-n convergence. Adv. Neural Inf. Process. Syst. 22, 117–125 (2009)

    Google Scholar 

  • Bura, E., Pfeiffer, R.M.: Graphical methods for class prediction using dimension reduction techniques on DNA microarray data. Bioinformatics 19(10), 1252–1258 (2003)

    Article  Google Scholar 

  • Chen, C.H.: Generalized association plots: information visualization via iteratively generated correlation matrices. Stat. Sin. 12, 7–29 (2002)

    MATH  Google Scholar 

  • Chen, C.H., Li, K.C.: Can SIR be as popular as multiple linear regression? Stat. Sin. 8, 289–316 (1998)

    MATH  Google Scholar 

  • Chen, C.H., Li, K.C.: Generalization of Fisher’s linear discriminant analysis via the approach of sliced inverse regression. J. Korean Stat. Soc. 30, 193–217 (2001)

    Google Scholar 

  • Coifman, R.R., Lafon, S., Lee, A.B., Maggioni, M., Nadler, B., Warner, F., Zucker, S.W.: Geometric diffusions as a tool for harmonic analysis and structure definition of data: diffusion maps. Proc. Natl. Acad. Sci. USA 102, 7426–7431 (2005)

    Article  Google Scholar 

  • Cook, R.D.: On the interpretation of regression plots. J. Am. Stat. Assoc. 89, 177–190 (1994)

    Article  MATH  Google Scholar 

  • Cook, R.D.: Graphics for regressions with a binary response. J. Am. Stat. Assoc. 91, 983–992 (1996)

    Article  MATH  Google Scholar 

  • Cook, R.D.: SAVE: a method for dimension reduction and graphics in regression. Commun. Stat., Theory Methods 29, 2109–2121 (2000)

    Article  MATH  Google Scholar 

  • Cook, R.D., Critchley, F.: Identifying regression outliers and mixtures graphically. J. Am. Stat. Assoc. 95, 781–794 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  • Cook, R.D., Ni, L.: Sufficient dimension reduction via inverse regression: a minimum discrepancy approach. J. Am. Stat. Assoc. 100(470), 410–428 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  • Cook, R.D., Ni, L.: Using intraslice covariances for improved estimation of the central subspace in regression. Biometrika 93(1), 65–74 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  • Cox, T.F., Cox, M.A.A.: Multidimensional Scaling. Chapman and Hall, London (1994)

    MATH  Google Scholar 

  • Dettling, M., Bühlmann, P.: Supervised clustering of genes. Genome Biol. 3(12), 0069 (2002)

    Article  Google Scholar 

  • Donoho, D.L., Grimes, C.: Hessian eigenmaps: locally linear embedding techniques for high-dimensional data. Proc. Natl. Acad. Sci. USA 100(10), 5591–5596 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  • Frank, A., Asuncion, A.: UCI machine learning repository. Irvine, CA: University of California, School of Information and Computer Science (2010). http://archive.ics.uci.edu/ml

  • Fukumizu, K., Bach, F.R., Jordan, M.I.: Kernel dimension reduction in regression. Ann. Stat. 37(4), 1871–1905 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  • Gaoa, X., Liang, J.: The dynamical neighborhood selection based on the sampling density and manifold curvature for isometric data embedding. Pattern Recognit. Lett. 32(2), 202–209 (2011)

    Article  Google Scholar 

  • Garber, M., et al.: Diversity of gene expression in adenocarcinoma of the lung. Proc. Natl. Acad. Sci. USA 98(24), 13784–13789 (2001)

    Article  Google Scholar 

  • Gather, U., Hilker, T., Becker, C.: A note on outlier sensitivity of sliced inverse regression. Statistics 36(4), 271–281 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  • Geng, X., Zhan, D.C., Zhou, Z.H.: Supervised nonlinear dimensionality reduction for visualization and classification. IEEE Trans. Syst. Man Cybern., Part B, Cybern. 35(6), 1098–1107 (2005)

    Article  Google Scholar 

  • Golub, T.R., et al.: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286(5439), 531–537 (1999)

    Article  Google Scholar 

  • Ham, J., Lee, D.D., Mika, S., Scholkopf, B.: A kernel view of the dimensionality reduction of manifolds. In: Proceedings of the Twenty-First International Conference on Machine Learning. ACM International Conference Proceeding Series, vol. 69 (2004)

    Google Scholar 

  • Hartigan, J.A., Wong, M.A.: A k-means clustering algorithm. Appl. Stat. 28, 100–108 (1979)

    Article  MATH  Google Scholar 

  • Hastie, T., Tibshirani, R.: Discriminant analysis by Gaussian mixtures. J. R. Stat. Soc. B 58, 155–176 (1996)

    MathSciNet  MATH  Google Scholar 

  • Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer, Berlin (2009)

    Book  Google Scholar 

  • Hsing, T.: Nearest neighbor inverse regression. Ann. Stat. 27(2), 697–731 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  • Khan, J., et al.: Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks. Nat. Med. 7, 673–679 (2001)

    Article  Google Scholar 

  • Kuss, M.: Nonlinear multivariate analysis with geodesic kernels. Technische Universitat Berlin, Diploma Theses (2002)

  • Lee, Y.J., Huang, S.Y.: Reduced support vector machines: a statistical theory. IEEE Trans. Neural Netw. 18, 1–13 (2007)

    Article  Google Scholar 

  • Li, K.C.: Sliced inverse regression for dimension reduction. J. Am. Stat. Assoc. 86, 316–342 (1991)

    Article  MATH  Google Scholar 

  • Li, L.: Sparse sufficient dimension reduction. Biometrika 94(3), 603–613 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  • Li, C.G., Guo, J.: Supervised isomap with explicit mapping. In: Proceedings of the First International Conference on Innovative Computing, Information and Control, vol. 3, pp. 345–348 (2006)

    Google Scholar 

  • Li, L., Yin, X.: Sliced inverse regression with regularizations. Biometrics 64(1), 124–131 (2007)

    Article  MathSciNet  Google Scholar 

  • Ni, L., Cook, R.D.: A robust inverse regression estimator. Stat. Probab. Lett. 77(3), 343–349 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  • Nilsson, J., Fioretos, T., Höglund, M., Fontes, M.: Approximate geodesic distances reveal biologically relevant structures in microarray data. Bioinformatics 20(6), 874–880 (2004)

    Article  Google Scholar 

  • Pomeroy, S.L., et al.: Prediction of central nervous system embryonal tumour outcome based on gene expression. Nature 415(24), 436–442 (2002)

    Article  Google Scholar 

  • Roweis, S., Saul, L.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000)

    Article  Google Scholar 

  • Samko, O., Marshall, A.D., Rosin, P.L.: Selection of the optimal parameter value for the ISOMAP algorithm. Pattern Recognit. Lett. 27(9), 968–979 (2006)

    Article  Google Scholar 

  • Saul, L.K., Roweis, S.T.: Think globally, fit locally: unsupervised learning of low dimensional manifolds. J. Mach. Learn. Res. 4, 119–155 (2003)

    MathSciNet  Google Scholar 

  • Setodji, C.M., Cook, R.D.: K-means inverse regression. Technometrics 46(4), 421–429 (2004)

    Article  MathSciNet  Google Scholar 

  • Singh, D., et al.: Gene expression correlates of clinical prostate cancer behavior. Cancer Cell 1(2), 203–209 (2002)

    Article  Google Scholar 

  • Smola, A.J., Schölkopf, B.: Sparse greedy matrix approximation for machine learning. In: Proceedings of the 17th International Conference on Machine Learning, Stanford University, CA, pp. 911–918. Morgan Kaufmann, San Mateo (2000)

    Google Scholar 

  • Schölkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge (2002)

    Google Scholar 

  • Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323 (2000)

    Article  Google Scholar 

  • Tien, Y.J., Lee, Y.S, Wu, H.M., Chen, C.H.: Methods for simultaneously identifying coherent local clusters with smooth global patterns in gene expression profiles. BMC Bioinform. 9, 155 (2008)

    Article  Google Scholar 

  • Vlachos, M., Domeniconi, C., Gunopulos, D., Kollios, G., Koudas, N.: Nonlinear dimensionality reduction techniques for classification and visualization. In: Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 645–651 (2002)

    Google Scholar 

  • Weinberger, K.Q., Sha, F., Saul, L.K.: Learning a kernel matrix for nonlinear dimensionality reduction. In: Proceedings of the Twenty First International Conference on Machine Learning (ICML 2004), Banff, Canada, pp. 839–846 (2004)

    Google Scholar 

  • Williams, C., Seeger, M.: Using the Nystrom method to speed up kernel machines. In: Leen, T.K., Dietterich, T.G., Tresp, V. (eds.) Advances in Neural Information Processing System, vol. 13, pp. 682–688. MIT Press, Cambridge (2001)

    Google Scholar 

  • Wu, H.M.: Kernel sliced inverse regression with applications on classification. J. Comput. Graph. Stat. 17(3), 590–610 (2008)

    Article  Google Scholar 

  • Wu, H.M., Lu, H.H.-S.: Supervised motion segmentation by spatial-frequential analysis and dynamic sliced inverse regression. Stat. Sin. 14, 413–430 (2004)

    MathSciNet  MATH  Google Scholar 

  • Wu, H.M., Lu, H.H.-S.: Iterative sliced inverse regression for segmentation of ultrasound and MR images. Pattern Recognit. 40(12), 3492–3502 (2007)

    Article  MATH  Google Scholar 

  • Wu, H.M., Tien, Y.J., Chen, C.H.: GAP: a graphical environment for matrix visualization and cluster analysis. Comput. Stat. Data Anal. 54, 767–778 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  • Wu, Q., Mukherjee, S., Liang, F.: Localized Sliced Inverse Regression. Advances in Neural Information Processing Systems, vol. 20. MIT Press, Cambridge (2008)

    Google Scholar 

  • Yeh, Y.R., Huang, S.Y., Lee, Y.J.: Nonlinear dimension reduction with kernel sliced inverse regression. IEEE Trans. Knowl. Data Eng. 21(11), 1590–1603 (2009)

    Article  Google Scholar 

  • Zhong, W., Zeng, P., Ma, P., Liu, J.S., Zhu, Y.: RSIR: regularized sliced inverse regression for motif discovery. Bioinformatics 21(22), 4169–4175 (2005)

    Article  Google Scholar 

Download references

Acknowledgements

The author thanks the associate editor and the referees for their valuable comments and suggestions. This research was supported by the grants from National Science Council at Taiwan, R.O.C. (NSC 99-2118-M-032-006-).

Author information

Authors and Affiliations

  1. Department of Mathematics, Tamkang University, New Taipei City, 25137, Taiwan, R.O.C.

    Wei-Ting Yao & Han-Ming Wu

Authors
  1. Wei-Ting Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Han-Ming Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Han-Ming Wu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Yao, WT., Wu, HM. Isometric sliced inverse regression for nonlinear manifold learning. Stat Comput 23, 563–576 (2013). https://doi.org/10.1007/s11222-012-9330-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-012-9330-z

Keywords

Search

Navigation