Abstract
The recognition of actions that involve the use of objects has remained a challenging task. In this paper, we present a hierarchical self-organizing neural architecture for learning to recognize transitive actions from RGB-D videos. We process separately body poses extracted from depth map sequences and object features from RGB images. These cues are subsequently integrated to learn action–object mappings in a self-organized manner in order to overcome the visual ambiguities introduced by the processing of body postures alone. Experimental results on a dataset of daily actions show that the integration of action–object pairs significantly increases classification performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Fleischer, F., Caggiano, V., Thier, P., Giese, M.A.: Physiologically inspired model for the visual recognition of transitive hand actions. J. Neurosci. 33(15), 6563–6580 (2013)
Nelissen, K., Luppino, G., Vanduffel, W., Rizzolatti, G., Orban, G.: Observing others: multiple action representation in frontal lobe. Science 310, 332–336 (2005)
Gallese, V., Fadiga, L., Fogassi, L., Rizzolatti, G.: Action recognition in premotor cortex. Brain 2, 593–609 (1996)
Saxe, R., Carey, S., Kanwisher, N.: Understanding other minds: linking developmental psychology and functional neuroimaging. Annu. Rev. Psychol. 55, 87–124 (2004)
Gupta, A., Davis, L.S.: Objects in action: an approach for combining action understanding and object perception. In: IEEE CVPR 2007, pp. 1–8 (2007)
Koppula, H.S., Gupta, R., Saxena, A.: Learning human activities and object affordances from RGB-D videos. Int. J. Robot. Res. 32(8), 951–970 (2013)
Yao, B., Fei-Fei, L.: Modeling mutual context of object and human pose in human-object interaction activities. In: IEEE CVPR 2010, pp. 17–24 (2010)
Kjellström, H., Romero, J., Kragíc, D.: Visual object-action recognition: inferring object affordances from human demonstration. Comput. Vis. Image Underst. 115(1), 81–90 (2011)
Mici, L., Hinaut, X., Wermter, S.: Activity recognition with echo state networks using 3D body joints and objects category. In: ESANN, Belgium, pp. 465–470 (2016)
Parisi, G.I., Weber, C., Wermter, S.: Self-organizing neural integration of pose-motion features for human action recognition. Front. Neurorobot. 9(3), 14 (2015)
Kohonen, T.: The self-organizing map. Proc. IEEE 78(9), 1464–1480 (1990)
Miikkulainen, R., Bednar, J.A., Choe, Y., Sirosh, J.: Computational Maps in the Visual Cortex. Springer Science & Business Media, New York (2006)
Evangelidis, G., Gurkirt, S., Radu, H.: Skeletal quads: human action recognition using joint quadruples. In: ICPR (2014)
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)
Jégou, H., Douze, M., Schmid, C., Pérez, P.: Aggregating local descriptors into a compact image representation. In: IEEE CVPR 2010, pp. 3304–3311 (2010)
Marsland, S., Shapiro, J., Nehmzow, U.: A self-organising network that grows when required. Neural Netw. 15(8), 1041–1058 (2002)
Fritzke, B.: A growing neural gas network learns topologies. Adv. Neural Inf. Process. Syst. 7, 625–632 (1995)
Acknowledgments
This research was partially supported by the Transregio TRR169 on Crossmodal Learning, by the DAAD German Academic Exchange Service for the Cognitive Assistive Systems project (Kz:A/13/94748), and the Hamburg Landesforschungsförderung.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Mici, L., Parisi, G.I., Wermter, S. (2016). Recognition of Transitive Actions with Hierarchical Neural Network Learning. In: Villa, A., Masulli, P., Pons Rivero, A. (eds) Artificial Neural Networks and Machine Learning – ICANN 2016. ICANN 2016. Lecture Notes in Computer Science(), vol 9887. Springer, Cham. https://doi.org/10.1007/978-3-319-44781-0_56
Download citation
DOI: https://doi.org/10.1007/978-3-319-44781-0_56
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-44780-3
Online ISBN: 978-3-319-44781-0
eBook Packages: Computer ScienceComputer Science (R0)