Abstract
In this work, we describe a classification of five sport-related human activities which are sensed by a thermal vision sensor. First, we have collected several sport sessions of an inhabitant while developing: push-ups, sit-ups, jumping jacks, squats and planks. Second, we develop an ad-hoc augmentation of data to increase the sturdiness of the data collection and reduce overfitting. Third, a Deep Learning model has been evaluated to compute a sequence of images from the user in order to estimate the activity. A CNN extracts relavant features from spatial domain and LSTM network models the sequence of images to compute the final classification. The results show an encouraging performance and quick learning capabilities.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Al-Sarawi, S., Anbar, M., Alieyan, K., Alzubaidi, M.: Internet of things (iot) communication protocols. In: 2017 8th International conference on information technology (ICIT), pp. 685–690. IEEE (2017)
Cireşan, D.C., Meier, U., Masci, J., Gambardella, L.M., Schmidhuber, J.: High-performance neural networks for visual object classification. arXiv preprint arXiv:1102.0183 (2011)
Dang, L.M., Min, K., Wang, H., Piran, M.J., Lee, C.H., Moon, H.: Sensor-based and vision-based human activity recognition: a comprehensive survey. Pattern Recogn. 108, 107561 (2020)
De-La-Hoz-Franco, E., Ariza-Colpas, P., Quero, J.M., Espinilla, M.: Sensor-based datasets for human activity recognition-a systematic review of literature. IEEE Access 6, 59192–59210 (2018)
Gochoo, M., Tan, T.H., Batjargal, T., Seredin, O., Huang, S.C.: Device-free non-privacy invasive indoor human posture recognition using low-resolution infrared sensor-based wireless sensor networks and dcnn. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 2311–2316. IEEE (2018)
Gochoo, M., et al.: Novel IoT-based privacy-preserving yoga posture recognition system using low-resolution infrared sensors and deep learning. IEEE Internet Things J. 6(4), 7192–7200 (2019)
Griffiths, E., Assana, S., Whitehouse, K.: Privacy-preserving image processing with binocular thermal cameras. Proc. ACM Interact. Mob. Wearable Ubiq. Technol. 1(4), 1–25 (2018)
Han, J., Bhanu, B.: Human activity recognition in thermal infrared imagery. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005)-Workshops, p. 17. IEEE (2005)
Hiriyannaiah, S., Akanksh, B.S., Koushik, A.S., Siddesh, G.M., Srinivasa, K.G.: Deep learning for multimedia data in IoT. In: Tanwar, S., Tyagi, S., Kumar, N. (eds.) Multimedia Big Data Computing for IoT Applications. ISRL, vol. 163, pp. 101–129. Springer, Singapore (2020). https://doi.org/10.1007/978-981-13-8759-3_4
Kong, X., Meng, Z., Meng, L., Tomiyama, H.: A privacy protected fall detection IoT system for elderly persons using depth camera. In: 2018 International Conference on Advanced Mechatronic Systems (ICAMechS), pp. 31–35. IEEE (2018)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1–9 (2012)
Martínez-González, A., Villamizar, M., Canévet, O., Odobez, J.M.: Efficient convolutional neural networks for depth-based multi-person pose estimation. IEEE Trans. Circ. Syst. Video Technol. 30(11), 4207–4221 (2019)
Medina-Quero, J., Zhang, S., Nugent, C., Espinilla, M.: Ensemble classifier of long short-term memory with fuzzy temporal windows on binary sensors for activity recognition. Expert Syst. Appl. 114, 441–453 (2018)
Nadeem, A., Jalal, A., Kim, K.: Automatic human posture estimation for sport activity recognition with robust body parts detection and entropy markov model. Multimedia Tools Appl. 80(14), 21465–21498 (2021). https://doi.org/10.1007/s11042-021-10687-5
Nasiri, S., Khosravani, M.R.: Progress and challenges in fabrication of wearable sensors for health monitoring. Sens. Actuators A: Phys. 312, 112105 (2020)
Ordóñez, F.J., Roggen, D.: Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1), 115 (2016)
Polo-Rodriguez, A., Vilchez Chiachio, J.M., Paggetti, C., Medina-Quero, J.: Ambient sound recognition of daily events by means of convolutional neural networks and fuzzy temporal restrictions. Appl. Sci. 11(15), 6978 (2021)
Ramasamy Ramamurthy, S., Roy, N.: Recent trends in machine learning for human activity recognition-a survey. Wiley Interdisc. Rev. Data Min. Knowl. Disc. 8(4), e1254 (2018)
Sixsmith, A., Johnson, N.: A smart sensor to detect the falls of the elderly. IEEE Perv. Comput. 3(2), 42–47 (2004)
Sozykin, K., Protasov, S., Khan, A., Hussain, R., Lee, J.: Multi-label class-imbalanced action recognition in hockey videos via 3d convolutional neural networks. In: 2018 19th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), pp. 146–151. IEEE (2018)
Wang, J., Chen, Y., Hao, S., Peng, X., Hu, L.: Deep learning for sensor-based activity recognition: a survey. Pattern Recogn. Lett. 119, 3–11 (2019)
Yadav, S.K., Tiwari, K., Pandey, H.M., Akbar, S.A.: A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions. Knowl.-Based Syst. 223, 106970 (2021)
Yamashita, T., Watasue, T., Yamauchi, Y., Fujiyoshi, H.: Improving quality of training samples through exhaustless generation and effective selection for deep convolutional neural networks. In: VISAPP, no. 2, pp. 228–235 (2015)
Zhang, C., Yang, F., Li, G., Zhai, Q., Jiang, Y., Xuan, D.: Mv-sports: a motion and vision sensor integration-based sports analysis system. In: IEEE INFOCOM 2018-IEEE Conference on Computer Communications, pp. 1070–1078. IEEE (2018)
Zhang, S., et al.: Deep learning in human activity recognition with wearable sensors: a review on advances. Sensors 22(4), 1476 (2022)
Zhang, S., Wei, Z., Nie, J., Huang, L., Wang, S., Li, Z.: A review on human activity recognition using vision-based method. J. Healthcare Eng. 2017 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Polo-Rodriguez, A., Montoro-Lendinez, A., Espinilla, M., Medina-Quero, J. (2022). Classifying Sport-Related Human Activity from Thermal Vision Sensors Using CNN and LSTM. In: Mazzeo, P.L., Frontoni, E., Sclaroff, S., Distante, C. (eds) Image Analysis and Processing. ICIAP 2022 Workshops. ICIAP 2022. Lecture Notes in Computer Science, vol 13373. Springer, Cham. https://doi.org/10.1007/978-3-031-13321-3_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-13321-3_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-13320-6
Online ISBN: 978-3-031-13321-3
eBook Packages: Computer ScienceComputer Science (R0)