Abstract
Auto-encoders are perhaps the best-known non-probabilistic methods for representation learning. They are conceptually simple and easy to train. Recent theoretical work has shed light on their ability to capture manifold structure, and drawn connections to density modeling. This has motivated researchers to seek ways of auto-encoder scoring, which has furthered their use in classification. Gated auto-encoders (GAEs) are an interesting and flexible extension of auto-encoders which can learn transformations among different images or pixel covariances within images. However, they have been much less studied, theoretically or empirically. In this work, we apply a dynamical systems view to GAEs, deriving a scoring function, and drawing connections to Restricted Boltzmann Machines. On a set of deep learning benchmarks, we also demonstrate their effectiveness for single and multi-label classification.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Bengio, Y., Thibodeau-Laufer, É.: Deep generative stochastic networks trainable by backprop (2013). arXiv preprint arXiv:1306.1091
Boutell, M.R., Luob, J., Shen, X., Brown, C.M.: Learning multi-label scene classification. Pattern Recognition 37, 1757–1771 (2004)
Cho, K.H., Ilin, A., Raiko, T.: Improved learning of Gaussian-Bernoulli restricted Boltzmann machines. In: Honkela, T. (ed.) ICANN 2011, Part I. LNCS, vol. 6791, pp. 10–17. Springer, Heidelberg (2011)
Droniou, A., Sigaud, O.: Gated autoencoders with tied input weights. In: ICML (2013)
Elisseeff, A., Weston, J.: A kernel method for multi-labelled classification. In: NIPS (2002)
Guillaume, A., Bengio, Y.: What regularized auto-encoders learn from the data generating distribution. In: ICLR (2013)
Kamyshanska, H., Memisevic, R.: On autoencoder scoring. In: ICML, pp. 720–728 (2013)
Kamyshanska, H., Memisevic, R.: The potential energy of an auto-encoder. IEEE Transactions on Pattern Analysis and Machine Intelligence 37(6), 1261–1273 (2014)
Krizhevsky, A.: Learning multiple layers of features from tiny images. Tech. rep., Department of Computer Science, University of Toronto (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., Bengio, Y.: An empirical evaluation of deep architectures on problems with many factors of variation. In: ICML (2007)
Li, Y., Tarlow, D., Zemel, R.: Exploring compositional high order pattern potentials for structured output learning. In: CVPR (2013)
Mandel, M.I., Eck, D., Bengio, Y.: Learning tags that vary within a song. In: ISMIR (2010)
Mandel, M.I., Ellis, D.P.W.: A web-based game for collecting music metadata. Journal New of Music Research 37, 151–165 (2008)
Memisevic, R.: Gradient-based learning of higher-order image features. In: ICCV (2011)
Memisevic, R., Zach, C., Hinton, G., Pollefeys, M.: Gated softmax classification. In: NIPS (2010)
Mnih, V., Hinton, G.E.: Learning to detect roads in high-resolution aerial images. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part VI. LNCS, vol. 6316, pp. 210–223. Springer, Heidelberg (2010)
Mnih, V., Larochelle, H., Hinton, G.E.: Conditional restricted Boltzmann machines for structured output prediction. In: UAI (2011)
Ranzato, M., Hinton, G.E.: Modeling pixel means and covariances using factorized third-order Boltzmann machines. In: CVPR (2010)
Rifai, S.: Contractive auto-encoders: explicit invariance during feature extraction. In: ICML (2011)
Swersky, K., Ranzato, M., Buchman, D., Freitas, N.D., Marlin, B.M.: On autoencoders and score matching for energy based models. In: ICML, pp. 1201–1208 (2011)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions (2014). arXiv preprint arXiv:1409.4842
Taylor, G.W., Hinton, G.E.: Factored conditional restricted Boltzmann machines for modeling motion style. In: ICML, pp. 1025–1032 (2009)
Vincent, P.: A connection between score matching and denoising auto-encoders. Neural Computation 23(7), 1661–1674 (2010)
Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.: Extracting and composing robust features with denoising autoencoders. In: ICML (2008)
Wang, N., Melchior, J., Wiskott, L.: Gaussian-binary restricted Boltzmann machines on modeling natural image statistics. Tech. rep., Institut fur Neuroinformatik Ruhr-Universitat Bochum, Bochum, 44780, Germany (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Im, D.J., Taylor, G.W. (2015). Scoring and Classifying with Gated Auto-Encoders. In: Appice, A., Rodrigues, P., Santos Costa, V., Soares, C., Gama, J., Jorge, A. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2015. Lecture Notes in Computer Science(), vol 9284. Springer, Cham. https://doi.org/10.1007/978-3-319-23528-8_33
Download citation
DOI: https://doi.org/10.1007/978-3-319-23528-8_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-23527-1
Online ISBN: 978-3-319-23528-8
eBook Packages: Computer ScienceComputer Science (R0)