Abstract
Anticipating salient events by actively predicting the sensory stream is a core skill of intelligent agents. In recent years, a variety of unsupervised deep learning approaches have been proposed for modeling our ability to predict the physical dynamics of visual scenes: In this paper we conduct a systematic evaluation of state-of-the-art models, considered capable of learning the spatio-temporal structure of synthetic videos of bouncing objects. We show that, though most of the models obtain high accuracy on the standard benchmark of predicting the next frame of a given sequence, they all fall short when probed with the generation of multiple future frames. Our simulations thus show that the ability to perform one-step-ahead prediction does not imply that the model has captured the underlying dynamics of the environment, suggesting that the gap between deep generative models and human observers has yet to be filled.
A. Cenzato and A. Testolin—Equal contribution.
This work was supported by the Cariparo Excellence Grant 2018 “Numsense” to M.Z., and by the Stars Grant “Deepmath” from the University of Padova to A.T.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
We call it “blind” because the model, when predicting frames \({\MakeUppercase {\hat{x}}}_{t+i}\) does not have any information about the ground-truth frames \({\MakeUppercase {x}}_{t+i}\).
- 3.
We compute the centroids of the connected components in each frame of the sequence, by first eroding each image and removing pixels under a fixed threshold. Erosion and threshold are applied to avoid that two colliding balls are detected as one single object.
- 4.
This interval allows to unroll the prediction for a sufficient amount of steps and it ensures that several collisions will occur (between the balls, or with the bounding box).
References
Berkes, P., Orbán, G., Lengyel, M., Fiser, J.: Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment. Science 331(6013), 83–87 (2011)
Cavanagh, P., Alvarez, G.A.: Tracking multiple targets with multifocal attention. Trends Cogn. Sci. 9(7), 349–354 (2005)
Chang, M.B., Ullman, T., Torralba, A., Tenenbaum, J.B.: A compositional object-based approach to learning physical dynamics. International Conference on Learning Representations (2017)
Clark, A.: Whatever next? predictive brains, situated agents, and the future of cognitive science. Behav. Brain Sci. 36(3), 181–204 (2013)
Donahue, J., et al.: Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition. pp. 2625–2634. (2015)
Finn, C., Goodfellow, I., Levine, S.: Unsupervised learning for physical interaction through video prediction. In: Advances in Neural Information Processing Systems. pp. 64–72 (2016)
Fiser, J., Aslin, R.N.: Unsupervised statistical learning of higher-order spatial structures from visual scenes. Psychol. Sci. 12(6), 499–504 (2001)
Friston, K.: The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11(2), 127 (2010)
Gan, Z., Li, C., Henao, R., Carlson, D. E., Carin, L.: Deep temporal sigmoid belief networks for sequence modeling. In: Advances in Neural Information Processing Systems. pp. 2467–2475 (2015)
Goodman, N. D., Tenenbaum, J. B., Contributors, T.P.: Probabilistic Models of Cognition (Second ed.). http://probmods.org/v2. Accessed 13 May 2019
Graves, A., Mohamed, A.-R., Hinton, G.: Speech recognition with deep recurrent neural networks. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 6645–6649 (2013)
Hinton, G.E.: Learning multiple layers of representation. Trends Cogn. Sci. 11(10), 428–434 (2007)
Hjelm, R.D., et al.: Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 (2018)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Jia, X., Hong, H., DiCarlo, J.J.: Unsupervised changes in core object recognition behavioral performance are accurately predicted by unsupervised neural plasticity in inferior temporal cortex. bioRxiv (2020)
Lotter, W., Kreiman, G., Cox, D.: Unsupervised learning of visual structure using predictive generative networks. arXiv preprint arXiv:1511.06380 (2015)
Lotter, W., Kreiman, G., Cox, D.: Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv:1605.08104 (2016)
Luc, P., et al.: Transformation-based adversarial video prediction on large-scale data. arXiv preprint arXiv:2003.04035 (2020)
Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440 (2015)
Oord, A.V.D., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
Ranzato, M., Szlam, A., Bruna, J., Mathieu, M., Collobert, R., Chopra, S.: Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv:1412.6604 (2014)
Srivastava, N., Mansimov, E., Salakhudinov, R.: Unsupervised learning of video representations using lstms. In: International Conference on Machine Learning. pp. 843–852 (2015)
Sutskever, I., Hinton, G.E., Taylor, G.W.: The recurrent temporal restricted boltzmann machine. In: Advances in Neural Information Processing Systems. pp. 1601–1608 (2009)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems. pp. 3104–3112 (2014)
Testolin, A., Stoianov, I., Sperduti, A., Zorzi, M.: Learning orthographic structure with sequential generative neural networks. Cogn. Sci. 40(3), 579–606 (2016)
Testolin, A., Zorzi, M.: Probabilistic models and generative neural networks: towards an unified framework for modeling normal and impaired neurocognitive functions. Front. Computat. Neurosci. 10, 73 (2016)
Tresilian, J.: Perceptual and cognitive processes in time-to-contact estimation: analysis of prediction-motion and relative judgment tasks. Perception Psychophys. 57(2), 231–245 (1995)
Villegas, R., Pathak, A., Kannan, H., Erhan, D., Le, Q. V., Lee, H.: High fidelity video prediction with large stochastic recurrent neural networks. In: Advances in Neural Information Processing Systems. pp. 81–91 (2019)
Villegas, R., Yang, J., Zou, Y., Sohn, S., Lin, X., Lee, H.: Learning to generate long-term future via hierarchical prediction. In: Proceedings of the 34th International Conference on Machine Learning. vol. 70, pp. 3560–3569 (2017)
Xingjian, S., et al.: Convolutional lstm network: a machine learning approach for precipitation nowcasting. In: Advances in Neural Information Processing Systems. pp. 802–810 (2015)
Xu, Z., et al.: Unsupervised discovery of parts, structure, and dynamics. In: International Conference on Learning Representations (2019)
Zaremba, W., Sutskever, I.: Learning to execute. arXiv preprint arXiv:1410.4615 (2014)
Zheng, D., Luo, V., Wu, J., D Tenenbaum, J. B.: Unsupervised learning of latent physical properties using perception-prediction networks. arXiv preprint arXiv:1807.09244 (2018)
Zhou, Y., Dong, H., El Saddik, A.: Deep learning in next-frame prediction: a benchmark review. IEEE Access 8, 69273–69283 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Cenzato, A., Testolin, A., Zorzi, M. (2020). Long-Term Prediction of Physical Interactions: A Challenge for Deep Generative Models. In: Nicosia, G., et al. Machine Learning, Optimization, and Data Science. LOD 2020. Lecture Notes in Computer Science(), vol 12565. Springer, Cham. https://doi.org/10.1007/978-3-030-64583-0_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-64583-0_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64582-3
Online ISBN: 978-3-030-64583-0
eBook Packages: Computer ScienceComputer Science (R0)