Skip to main content
SpringerLink
Log in

Disentanglement and Local Directions of Variance

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases. Research Track (ECML PKDD 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12977))

  • 1743 Accesses

Abstract

Previous line of research on learning disentangled representations in an unsupervised setting focused on enforcing an uncorrelated posterior. These approaches have been shown both empirically and theoretically to be insufficient for guaranteeing disentangled representations. Recent works postulate that an implicit PCA-like behavior might explain why these models still tend to disentangle, exploiting the structure of variance in the datasets. Here we aim to further verify those hypotheses by conducting multiple analyses on existing benchmark datasets and models, focusing on the relation between the structure of variance induced by the ground-truth factors and properties of the learned representations. We quantify the effects of global and local directions of variance in the data on disentanglement performance using proposed measures and seem to find empirical evidence of a negative effect of local variance directions on disentanglement. We also invalidate the robustness of models with a global ordering of latent dimensions against the local vs. global discrepancies in the data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    github.com/google-research/disentanglement_lib.

References

  1. Baldi, P., Hornik, K.: Neural networks and principal component analysis: learning from examples without local minima. Neural Netw. 2(1), 53–58 (1989)

    Article  Google Scholar 

  2. Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013)

    Article  Google Scholar 

  3. Bouchacourt, D., Tomioka, R., Nowozin, S.: Multi-level variational autoencoder: learning disentangled representations from grouped observations. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

  4. Bourlard, H., Kamp, Y.: Auto-association by multilayer perceptrons and singular value decomposition. Biol. Cybern. 59(4), 291–294 (1988)

    Article  MathSciNet  Google Scholar 

  5. Burgess, C.P., et al.: Understanding disentangling in beta-VAE. arXiv preprint arXiv:1804.03599 (2018)

  6. Chen, T.Q., Li, X., Grosse, R.B., Duvenaud, D.K.: Isolating sources of disentanglement in variational autoencoders. In: Advances in Neural Information Processing Systems, pp. 2610–2620 (2018)

    Google Scholar 

  7. Dai, B., Wang, Y., Aston, J., Hua, G., Wipf, D.: Connections with robust PCA and the role of emergent sparsity in variational autoencoder models. J. Mach. Learn. Res. 19(1), 1573–1614 (2018)

    MathSciNet  MATH  Google Scholar 

  8. Duan, S., et al.: Unsupervised model selection for variational disentangled representation learning. In: International Conference on Learning Representations (2019)

    Google Scholar 

  9. Eastwood, C., Williams, C.K.: A framework for the quantitative evaluation of disentangled representations (2018)

    Google Scholar 

  10. Eriksson, J., Koivunen, V.: Identifiability and separability of linear ICA models revisited. In: Proceedings of ICA, vol. 2003, pp. 23–27 (2003)

    Google Scholar 

  11. Gondal, M.W., et al.: On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. Adv. Neural Inf. Process. Syst. 32, 15740–15751 (2019)

    Google Scholar 

  12. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)

    MATH  Google Scholar 

  13. Higgins, I., et al.: Early visual concept learning with unsupervised deep learning. arXiv preprint arXiv:1606.05579 (2016)

  14. Hosoya, H.: Group-based learning of disentangled representations with generalizability for novel contents. In: IJCAI, pp. 2506–2513 (2019)

    Google Scholar 

  15. Hyvärinen, A.: Survey on independent component analysis (1999)

    Google Scholar 

  16. Khemakhem, I., Kingma, D., Monti, R., Hyvarinen, A.: Variational autoencoders and nonlinear ICA: a unifying framework. In: International Conference on Artificial Intelligence and Statistics, pp. 2207–2217. PMLR (2020)

    Google Scholar 

  17. Kim, H., Mnih, A.: Disentangling by factorising. In: International Conference on Machine Learning, pp. 2649–2658. PMLR (2018)

    Google Scholar 

  18. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 (2013)

  19. Kumar, A., Sattigeri, P., Balakrishnan, A.: Variational inference of disentangled latent concepts from unlabeled observations. In: International Conference on Learning Representations (2018)

    Google Scholar 

  20. Kunin, D., Bloom, J., Goeva, A., Seed, C.: Loss landscapes of regularized linear autoencoders. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 3560–3569. PMLR, 09–15 June 2019

    Google Scholar 

  21. LeCun, Y., Huang, F.J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2004. CVPR 2004. vol. 2, pp. II–104. IEEE (2004)

    Google Scholar 

  22. Locatello, F., et al.: Challenging common assumptions in the unsupervised learning of disentangled representations. In: International Conference on Machine Learning, pp. 4114–4124. PMLR (2019)

    Google Scholar 

  23. Locatello, F., et al.: A commentary on the unsupervised learning of disentangled representations. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 13681–13684 (2020)

    Google Scholar 

  24. Locatello, F., et al.: Weakly-supervised disentanglement without compromises. In: International Conference on Machine Learning, pp. 6348–6359. PMLR (2020)

    Google Scholar 

  25. Lucas, J., Tucker, G., Grosse, R.B., Norouzi, M.: Don’t blame the ELBO! A linear VAE perspective on posterior collapse. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019)

    Google Scholar 

  26. Pandey, A., Fanuel, M., Schreurs, J., Suykens, J.A.: Disentangled representation learning and generation with manifold optimization. arXiv preprint arXiv:2006.07046 (2020)

  27. Pearson, K.: LIII. On lines and planes of closest fit to systems of points in space. London Edinburgh Dublin Philos. Mag. J. Sci. 2(11), 559–572 (1901)

    Article  Google Scholar 

  28. Peters, J., Janzing, D., Schölkopf, B.: Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, Cambridge (2017)

    MATH  Google Scholar 

  29. Plaut, E.: From principal subspaces to principal components with linear autoencoders. arXiv preprint arXiv:1804.10253 (2018)

  30. Reed, S.E., Zhang, Y., Zhang, Y., Lee, H.: Deep visual analogy-making. In: Advances in Neural Information Processing Systems, pp. 1252–1260 (2015)

    Google Scholar 

  31. Ren, X., Yang, T., Wang, Y., Zeng, W.: Rethinking content and style: exploring bias for unsupervised disentanglement. arXiv preprint arXiv:2102.10544 (2021)

  32. Ridgeway, K., Mozer, M.C.: Learning deep disentangled embeddings with the f-statistic loss. In: Advances in Neural Information Processing Systems, pp. 185–194 (2018)

    Google Scholar 

  33. Rolinek, M., Zietlow, D., Martius, G.: Variational autoencoders pursue PCA directions (by accident). In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12406–12415 (2019)

    Google Scholar 

  34. Schölkopf, B., Smola, A., Müller, K.-R.: Kernel principal component analysis. In: Gerstner, W., Germond, A., Hasler, M., Nicoud, J.-D. (eds.) ICANN 1997. LNCS, vol. 1327, pp. 583–588. Springer, Heidelberg (1997). https://doi.org/10.1007/BFb0020217

    Chapter  Google Scholar 

  35. Scholz, M., Vigário, R.: Nonlinear PCA: a new hierarchical approach. In: ESANN, pp. 439–444 (2002)

    Google Scholar 

  36. Shu, R., Chen, Y., Kumar, A., Ermon, S., Poole, B.: Weakly supervised disentanglement with guarantees. In: International Conference on Learning Representations (2019)

    Google Scholar 

  37. Stühmer, J., Turner, R., Nowozin, S.: Independent subspace analysis for unsupervised learning of disentangled representations. In: International Conference on Artificial Intelligence and Statistics, pp. 1200–1210. PMLR (2020)

    Google Scholar 

  38. Suykens, J.A.: Deep restricted kernel machines using conjugate feature duality. Neural Comput. 29(8), 2123–2163 (2017)

    Article  MathSciNet  Google Scholar 

  39. Tipping, M.E., Bishop, C.M.: Probabilistic principal component analysis. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 61(3), 611–622 (1999)

    Article  MathSciNet  Google Scholar 

  40. Tonin, F., Patrinos, P., Suykens, J.A.: Unsupervised learning of disentangled representations in deep restricted kernel machines with orthogonality constraints. arXiv preprint arXiv:2011.12659 (2020)

  41. Zietlow, D., Rolinek, M., Martius, G.: Demystifying inductive biases for \(beta\)-VAE based architectures. arXiv preprint arXiv:2102.06822 (2021)

Download references

Author information

Authors and Affiliations

  1. Hasso Plattner Institute for Digital Engineering, University of Potsdam, Potsdam, Germany

    Alexander Rakowski & Christoph Lippert

  2. Hasso Plattner Institute for Digital Health at Mount Sinai, New York, USA

    Christoph Lippert

Authors
  1. Alexander Rakowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christoph Lippert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexander Rakowski .

Editor information

Editors and Affiliations

  1. ELLIS - The European Laboratory for Learning and Intelligent Systems, Alicante, Spain

    Nuria Oliver

  2. ETHZ and EPFL, Zürich, Switzerland

    Fernando Pérez-Cruz

  3. Johannes Gutenberg University of Mainz, Mainz, Germany

    Stefan Kramer

  4. École Polytechnique, Palaiseau, France

    Jesse Read

  5. Basque Center for Applied Mathematics, Bilbao, Spain

    Jose A. Lozano

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rakowski, A., Lippert, C. (2021). Disentanglement and Local Directions of Variance. In: Oliver, N., Pérez-Cruz, F., Kramer, S., Read, J., Lozano, J.A. (eds) Machine Learning and Knowledge Discovery in Databases. Research Track. ECML PKDD 2021. Lecture Notes in Computer Science(), vol 12977. Springer, Cham. https://doi.org/10.1007/978-3-030-86523-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86523-8_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86522-1

  • Online ISBN: 978-3-030-86523-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation