Abstract
Human Activity Recognition is a field that provides the fundamentals for Ambient Intelligence and Assisted Living Applications. Multimodal methods for Human Activity Recognition utilize different sensors and fuse them together to provide higher-accuracy results. These methods require data for all sensors employed to operate with. In this work we present a sensor-independent, in regards to the number of sensors used, scheme for designing multimodal methods that operate when sensor-data are missing. Furthermore, we present a data augmentation method that increases the fusion model’s accuracy (up to \(11\%\) increases) when operating with missing sensor-data. The proposed method’s effectiveness is evaluated on the ExtraSensory dataset, which contains over 300,000 samples from 60 users, collected from smartphones and smartwatches. In addition, the methods are evaluated for different number of sensors used at the same time. However, the max number of sensors must be known beforehand.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aggarwal, J.K., Xia, L.: Human activity recognition from 3d data: a review. Pattern Recogn. Lett. 48, 70–80 (2014)
Barrios-Avilés, J., Iakymchuk, T., Samaniego, J., Medus, L.D., Rosado-Muñoz, A.: Movement detection with event-based cameras: comparison with frame-based cameras in robot object tracking using powerlink communication. Electronics 7(11), 304 (2018)
Batchuluun, G., Nguyen, D.T., Pham, T.D., Park, C., Park, K.R.: Action recognition from thermal videos. IEEE Access 7, 103893–103917 (2019)
Chandrasekaran, B., Gangadhar, S., Conrad, J.M.: A survey of multisensor fusion techniques, architectures and methodologies. In: SoutheastCon 2017, pp. 1–8. IEEE (2017)
Dong, Y., Li, X., Dezert, J., Khyam, M.O., Noor-A-Rahim, M., Ge, S.S.: Dezert-Smarandache theory-based fusion for human activity recognition in body sensor networks. IEEE Trans. Industr. Inf. 16(11), 7138–7149 (2020)
Ehatisham-Ul-Haq, M.: Robust human activity recognition using multimodal feature-level fusion. IEEE Access 7, 60736–60751 (2019)
Grabisch, M., Raufaste, E.: An empirical study of statistical properties of the Choquet and Sugeno integrals. IEEE Trans. Fuzzy Syst. 16(4), 839–850 (2008)
Innocenti, S.U., Becattini, F., Pernici, F., Del Bimbo, A.: Temporal binary representation for event-based action recognition. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 10426–10432. IEEE (2021)
Lee, Y.-S., Cho, S.-B.: Activity recognition using hierarchical hidden Markov models on a smartphone with 3d accelerometer. In: Corchado, E., Kurzyński, M., Woźniak, M. (eds.) HAIS 2011. LNCS (LNAI), vol. 6678, pp. 460–467. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21219-2_58
Li, H., et al.: Multisensor data fusion for human activities classification and fall detection. In: 2017 IEEE SENSORS, pp. 1–3. IEEE (2017)
Naik, K., Pandit, T., Naik, N., Shah, P.: Activity recognition in residential spaces with internet of things devices and thermal imaging. Sensors 21(3), 988 (2021)
Nweke, H.F., Teh, Y.W., Mujtaba, G., Al-Garadi, M.A.: Data fusion and multiple classifier systems for human activity detection and health monitoring: review and open research directions. Inf. Fusion 46, 147–170 (2019)
Ordóñez, F.J., Roggen, D.: Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1), 115 (2016)
Sebestyen, G., Stoica, I., Hangan, A.: Human activity recognition and monitoring for elderly people. In: 2016 IEEE 12th International Conference on Intelligent Computer Communication and Processing (ICCP), pp. 341–347. IEEE (2016)
Uddin, M.Z., Soylu, A.: Human activity recognition using wearable sensors, discriminant analysis, and long short-term memory-based neural structured learning. Sci. Rep. 11(1), 16455 (2021)
Vaizman, Y., Ellis, K., Lanckriet, G.: Recognizing detailed human context in the wild from smartphones and smartwatches. IEEE Pervasive Comput. 16(4), 62–74 (2017). https://doi.org/10.1109/MPRV.2017.3971131
Vrigkas, M., Nikou, C., Kakadiaris, I.A.: A review of human activity recognition methods. Front. Robot. AI 2, 28 (2015). https://doi.org/10.3389/frobt.2015.00028
Wang, L., Huynh, D.Q., Koniusz, P.: A comparative review of recent kinect-based action recognition algorithms. IEEE Trans. Image Process. 29, 15–28 (2019)
Wu, Q., Wang, Z., Deng, F., Chi, Z., Feng, D.D.: Realistic human action recognition with multimodal feature selection and fusion. IEEE Trans. Syst. Man Cybern. Syst. 43(4), 875–885 (2013). https://doi.org/10.1109/TSMCA.2012.2226575
Yao, S., Hu, S., Zhao, Y., Zhang, A., Abdelzaher, T.: DeepSense: a unified deep learning framework for time-series mobile sensing data processing. In: Proceedings of the 26th International Conference on World Wide Web, pp. 351–360 (2017)
Zeng, Z., Zhang, Z., Pianfetti, B., Tu, J., Huang, T.S.: Audio-visual affect recognition in activation-evaluation space. In: 2005 IEEE International Conference on Multimedia and Expo, p. 4. IEEE (2005)
Zhang, S., Wei, Z., Nie, J., Huang, L., Wang, S., Li, Z.: A review on human activity recognition using vision-based method. J. Healthc. Eng. 2017, 3090343 (2017)
Zhu, C., Sheng, W.: Multi-sensor fusion for human daily activity recognition in robot-assisted living. In: Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction, pp. 303–304 (2009)
Acknowledgments
The research work was supported by the Hellenic Foundation for Research and Innovation (H.F.R.I.) under the “First Call for H.F.R.I. Research Projects to support Faculty members and Researchers and the procurement of high-cost research equipment grant” (Project Name: ACTIVE, Project Number: HFRI-FM17-2271).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Alexiadis, A., Nizamis, A., Giakoumis, D., Votis, K., Tzovaras, D. (2022). A Sensor-Independent Multimodal Fusion Scheme for Human Activity Recognition. In: El Yacoubi, M., Granger, E., Yuen, P.C., Pal, U., Vincent, N. (eds) Pattern Recognition and Artificial Intelligence. ICPRAI 2022. Lecture Notes in Computer Science, vol 13364. Springer, Cham. https://doi.org/10.1007/978-3-031-09282-4_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-09282-4_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-09281-7
Online ISBN: 978-3-031-09282-4
eBook Packages: Computer ScienceComputer Science (R0)