Abstract
The recent progress on capsule networks by Hinton et al. has generated considerable excitement in the machine learning community. The idea behind a capsule is inspired by a cortical minicolumn in the brain, whereby a vertically organised group of around 100 neurons receive common inputs, have common outputs, are interconnected, and may well constitute a fundamental computation unit of the cerebral cortex. However, Hinton’s paper on “Matrix Capsule with EM Routing” was unfortunately not accompanied by a release of source code, which left interested researchers attempting to implement the architecture and reproduce the benchmarks on their own. This has certainly slowed the progress of research building on this work. While writing our own implementation, we noticed several common mistakes in other open source implementations that we came across. In this paper we share some of these learnings, specifically focusing on three implementation pitfalls and how to avoid them: (1) parent capsules with only one child; (2) normalising the amount of data assigned to parent capsules; (3) parent capsules at different positions compete for child capsules. While our implementation is a considerable improvement over currently available implementations, it still falls slightly short of the performance reported by Hinton et al. (2018). The source code for this implementation is available on GitHub at the following URL: https://github.com/IBM/matrix-capsules-with-em-routing.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Calvano, G.S.: Some clarification on the convolution topology? July 2018. https://openreview.net/forum?id=HJWLfGWRb¬eId=BJgX7Iy04m
Cruz, L., et al.: A statistically based density map method for identification and quantification of regional differences in microcolumnarity in the monkey brain. J. Neurosci. Methods 141(2), 321–332 (2005)
Hinton, G., Krizhevsky, A., Jaitly, N., Tieleman, T., Tang, Y.: Does the brain do inverse graphics? In: Brain and Cognitive Sciences Fall Colloquium, vol. 2 (2012)
Hinton, G.E., Krizhevsky, A., Wang, S.D.: Transforming auto-encoders. In: Honkela, T., Duch, W., Girolami, M., Kaski, S. (eds.) ICANN 2011. LNCS, vol. 6791, pp. 44–51. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21735-7_6
Hinton, G.E., Sabour, S., Frosst, N.: Lambda and margin, July 2018. https://openreview.net/forum?id=HJWLfGWRb¬eId=BkelcSxC47
Hinton, G.E., Sabour, S., Frosst, N.: Matrix capsules with EM routing. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=HJWLfGWRb
Hinton, G.E., Sabour, S., Frosst, N.: Regularization and learning rate? October 2018. https://openreview.net/forum?id=HJWLfGWRb¬eId=rJeQnSsE3X
Huang, Y.: Capsules. GitHub, April 2019. https://github.com/Officium/Capsules. Commit: e1f02d3
LeCun, Y., Cortes, C., Burges, C.J.: The MNIST database of handwritten digits (1998)
LeCun, Y., Huang, F.J., Bottou, L., et al.: Learning methods for generic object recognition with invariance to pose and lighting. In: CVPR, vol. 2, pp. 97–104. Citeseer (2004)
Lei, J.Y.: Matrix-capsules-em-pytorch. GitHub, March 2019. https://github.com/yl-1993/Matrix-Capsules-EM-PyTorch. Commit: c4547bf
Sabour, S.: Code for capsule model used in dynamic routing between capsules. GitHub, January 2018. https://github.com/Sarasra/models/tree/master/research/ capsules. Commit: cac8804
Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Advances in Neural Information Processing Systems, pp. 3856–3866 (2017)
Zhang, S.: Matrix-capsules-em-tensorflow. GitHub, February 2018. https://github.com/www0wwwjs1/Matrix-Capsules-EM-Tensorow. Commit: 0196ead
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Gritzman, A.D. (2019). Avoiding Implementation Pitfalls of “Matrix Capsules with EM Routing” by Hinton et al.. In: Zeng, A., Pan, D., Hao, T., Zhang, D., Shi, Y., Song, X. (eds) Human Brain and Artificial Intelligence. HBAI 2019. Communications in Computer and Information Science, vol 1072. Springer, Singapore. https://doi.org/10.1007/978-981-15-1398-5_16
Download citation
DOI: https://doi.org/10.1007/978-981-15-1398-5_16
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-1397-8
Online ISBN: 978-981-15-1398-5
eBook Packages: Computer ScienceComputer Science (R0)