Abstract
One of the major challenges in Human Activity Recognition (HAR) using cameras, is occlusion of one or more body parts. However, this problem is often underestimated in contemporary research works, wherein training and evaluation is based on datasets shot under laboratory conditions, i.e., without some kind of occlusion. In this work we propose an approach for HAR in the presence of partial occlusion, i.e., in case of up to two occluded body parts. We solve this problem using regression, performed by a deep neural network. That is, given an occluded sample, we attempt to reconstruct the missing information regarding the motion of the occluded part(s). We evaluate our approach using a publicly available human motion dataset. Our experimental results indicate a significant increase of performance, when compared to a baseline approach, wherein a network that has been trained using non-occluded samples is evaluated using occluded samples. To the best of our knowledge, this is the first research work that tackles the problem of HAR under occlusion as a regression problem.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abadi, M., et al.: TensorFlow: a system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2016) (2016)
Angelini, F., Fu, Z., Long, Y., Shao, L., Naqvi, S.M.: 2D pose-based real-time human action recognition with occlusion-handling. IEEE Trans. Multimed. 22(6), 1433–1446 (2019)
Antoshchuk, S., Kovalenko, M., Sieck, J.: Gesture recognition-based human–computer interaction interface for multimedia applications. In: Jat, D.S., Sieck, J., Muyingi, H.N.S.-N., Winschiers-Theophilus, H., Peters, A., Nggada, S. (eds.) Digitisation of Culture: Namibian and International Perspectives, pp. 269–286. Springer, Singapore (2018). https://doi.org/10.1007/978-981-10-7697-8_16
Chollet, F., et al.: Keras (2015). https://github.com/fchollet/keras
Debes, C., Merentitis, A., Sukhanov, S., Niessen, M., Frangiadakis, N., Bauer, A.: Monitoring activities of daily living in smart homes: understanding human behavior. IEEE Sign. Process. Mag. 33(2), 81–94 (2016)
Du, Y., Fu, Y., Wang, L.: Skeleton based action recognition with convolutional neural network. In: 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pp. 579–583. IEEE (2015)
Giannakos, I., Mathe, E., Spyrou, E., Mylonas, P.: A study on the effect of occlusion in human activity recognition. In: The 14th PErvasive Technologies Related to Assistive Environments Conference, pp. 473–482 (2021)
Gu, R., Wang, G., Hwang, J.N.: Exploring severe occlusion: multi-person 3D pose estimation with gated convolution. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 8243–8250. IEEE (2021)
Hou, Y., Li, Z., Wang, P., Li, W.: Skeleton optical spectra-based action recognition using convolutional neural networks. IEEE Trans. Circ. Syst. Video Technol. 28(3), 807–811 (2016)
Iosifidis, A., Tefas, A., Pitas, I.: Multi-view human action recognition under occlusion based on fuzzy distances and neural networks. In: Proceedings of the 20th European Signal Processing Conference (EUSIPCO). IEEE (2012)
Ke, Q., An, S., Bennamoun, M., Sohel, F., Boussaid, F.: SkeletonNet: mining deep part features for 3-D action recognition. IEEE Sign. Process. Lett. 24(6), 731–735 (2017)
Keogh, A., Dorn, J.F., Walsh, L., Calvo, F., Caulfield, B.: Comparing the usability and acceptability of wearable sensors among older Irish adults in a real-world context: observational study. JMIR Mhealth Uhealth 8(4), e15704 (2020)
Lawton, M.P., Brody, E.M.: Assessment of older people: self-maintaining and instrumental activities of daily living. Gerontol. 9(3 Part 1), 179–186 (1969)
Li, C., Hou, Y., Wang, P., Li, W.: Joint distance maps based action recognition with convolutional neural networks. IEEE Sign. Process. Lett. 24(5), 624–628 (2017)
Liu, M., Liu, H., Chen, C.: Enhanced skeleton visualization for view invariant human action recognition. Pattern Recogn. 68, 346–362 (2017)
Liu, C., Hu, Y., Li, Y., Song, S., Liu, J.: PKU-MMD: a large scale benchmark for continuous multi-modal human action understanding. arXiv preprint arXiv:1703.07475 (2017)
Liu, T., et al.: View-invariant, occlusion-robust probabilistic embedding for human pose. Int. J. Comput. Vis. 130(1), 111–135 (2022)
Majumder, S., Mondal, T., Deen, M.J.: Wearable sensors for remote health monitoring. Sensors 17(1), 130 (2017)
Papadakis, A., Mathe, E., Spyrou, E., Mylonas, P.: A geometric approach for cross-view human action recognition using deep learning. In: 11th International Symposium on Image and Signal Processing and Analysis (ISPA). IEEE (2019)
Ranasinghe, S., Al Machot, F., Mayr, H.C.: A review on applications of activity recognition systems with regard to performance and evaluation. Int. J. Distrib. Sens. Netw. 12(8), 1550147716665520 (2016)
Vernikos, I., Mathe, E., Papadakis, A., Spyrou, E., Mylonas, P.: An image representation of skeletal data for action recognition using convolutional neural networks. In: Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments, pp. 325–326, June 2019
Wang, P., Li, W., Li, C., Hou, Y.: Action recognition based on joint trajectory maps with convolutional neural networks. Knowl.-Based Syst. 158, 43–53 (2018)
Wang, P., Li, W., Ogunbona, P., Wan, J., Escalera, S.: RGB-D-based human motion recognition with deep learning: a survey. Comput. Vis. Image Underst. 171, 118–139 (2018)
Acknowledgements
This research was co-financed by the European Union and Greek national funds through the Competitiveness, Entrepreneurship and Innovation Operational Programme, under the Call «Special Actions “Aquaculture” - “Industrial materials” - “Open innovation in culture”»; project title: “Strengthening User Experience & Cultural Innovation through Experiential Knowledge Enhancement with Enhanced Reality Technologies - MON REPO”; project code: \(T6YB\varPi \) - 00303; MIS code: 5066856
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Kostis, IA., Mathe, E., Spyrou, E., Mylonas, P. (2022). Human Activity Recognition Under Partial Occlusion. In: Iliadis, L., Jayne, C., Tefas, A., Pimenidis, E. (eds) Engineering Applications of Neural Networks. EANN 2022. Communications in Computer and Information Science, vol 1600. Springer, Cham. https://doi.org/10.1007/978-3-031-08223-8_25
Download citation
DOI: https://doi.org/10.1007/978-3-031-08223-8_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-08222-1
Online ISBN: 978-3-031-08223-8
eBook Packages: Computer ScienceComputer Science (R0)