F. Xavier Gaya-Morey
Cristina Manresa-Yee
ABSTRACT
In this work, we present the recognition of daily activities of elderly people at their home environment, as a way to help this collective preserve their autonomy. To do so, we have taken three different neural networks with near state-of-the-art performance for human activity recognition and trained them on a big dataset of elderly people activities at home environments: ETRI-Activity3D. Then, we have explained the results adapting LIME to work with video input, to further validate the models and to help future users understand the models’ predictions. In addition, we have separated the explanations in space and time as a way of reducing the required computation time. Our main objective is that through the deployment of our work on a real environment, abnormal (such as repetitions or omissions) or alarming (like falls) activities can be discovered and help elderly people accordingly.
- 2021. World Health Organization (WHO) fact sheets: Falls.Google Scholar
- Riyad Al-Shaqi, Monjur Mourshed, and Yacine Rezgui. 2016. Progress in ambient assisted systems for independent living by the elderly. SpringerPlus 5, 1 (14 May 2016), 624.Google ScholarCross Ref
- Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58 (2020), 82–115.Google ScholarDigital Library
- Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021. Is Space-Time Attention All You Need for Video Understanding?Google Scholar
- D.E. Bloom and D.L. Luca. 2016. Chapter 1 - The Global Demography of Aging: Facts, Explanations, Future. Handbook of the Economics of Population Aging, Vol. 1. North-Holland, 3–56.Google Scholar
- D E Bloom and D L Luca. 2016. Chapter 1 - The Global Demography of Aging: Facts, Explanations, Future. Vol. 1. North-Holland, 3–56.Google Scholar
- Marco Buzzelli, Alessio Albé, and Gianluigi Ciocca. 2020. A vision-based system for monitoring elderly people at home. Applied Sciences (Switzerland) 10 (1 2020). Issue 1.Google Scholar
- Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. 2018. A Short Note about Kinetics-600.Google Scholar
- MMAction2 Contributors. 2020. OpenMMLab’s Next Generation Video Understanding Toolbox and Benchmark. https://github.com/open-mmlab/mmaction2Google Scholar
- A Costa, E Martinez-Martin, M Cazorla, and V Julian. 2018. PHAROS-PHysical Assistant RObot System. Sensors (Basel) 18 (8 2018). Issue 8.Google Scholar
- Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, and Trevor Darrell. 2017. Long-Term Recurrent Convolutional Networks for Visual Recognition and Description. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 4 (2017), 677–691.Google ScholarDigital Library
- Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. 2021. Multiscale Vision Transformers.Google Scholar
- Srujana Gattupalli, Dylan Ebert, Michalis Papakostas, Fillia Makedon, and Vassilis Athitsos. 2017. CogniLearn: A Deep Learning-Based Interface for Cognitive Behavior Assessment. Proceedings of the 22nd International Conference on Intelligent User Interfaces, 577–587.Google ScholarDigital Library
- Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyńska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. 2017. The "something something" video database for learning and evaluating visual common sense.Google Scholar
- Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. 2017. AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions.Google Scholar
- Liam Hiley, Alun Preece, and Yulia Hicks. 2019. Explainable Deep Learning for Video Recognition Tasks: A Framework & Recommendations.Google Scholar
- Jinhyeok Jang, Dohyung Kim, Cheonshu Park, Minsu Jang, Jaeyeon Lee, and Jaehong Kim. 2020. ETRI-Activity3D: A Large-Scale RGB-D Dataset for Robots to Recognize Daily Activities of the Elderly.Google Scholar
- Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The Kinetics Human Action Video Dataset.Google Scholar
- Elham Khodabandehloo, Daniele Riboni, and Abbas Alimohammadi. 2021. HealthXAI: Collaborative and explainable AI for supporting early diagnosis of cognitive decline. Future Generation Computer Systems 116 (2021), 168–189.Google ScholarCross Ref
- Lachaux Killian, Maitre Julien, Bouchard Kevin, Lussier Maxime, Bottari Carolina, Couture Mélanie, Bier Nathalie, Giroux Sylvain, and Gaboury Sebastien. 2021. Fall Prevention and Detection in Smart Homes Using Monocular Cameras and an Interactive Social Robot. Proceedings of the Conference on Information Technology for Social Good, 7–12.Google ScholarDigital Library
- Jong-Wook Kim, Young-Lim Choi, Sang-Hyun Jeong, and Jeonghye Han. 2022. A care robot with ethical sensing system for older adults at home. Sensors (Basel) 22 (10 2022), 7515. Issue 19.Google Scholar
- Pang Wei Koh and Percy Liang. 2017. Understanding Black-box Predictions via Influence Functions.Google Scholar
- Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu. 2020. TAM: Temporal Adaptive Module for Video Recognition. arXiv preprint arXiv:2005.06803 (2020).Google Scholar
- Kun-Lin Lu and Edward T.-H. Chu. 2018. An Image-Based Fall Detection System for the Elderly. Applied Sciences 8, 10 (2018).Google Scholar
- Sumit Majumder, Emad. Aghayi, Moein Noferesti, Hamidreza Memarzadeh-Tehran, Tapas Mondal, Zhibo Pang, and M. Jamal Deen. 2017. Smart Homes for Elderly Healthcare—Recent Advances and Research Challenges. Sensors 17, 11 (2017).Google Scholar
- Haider Mshali, Tayeb Lemlouma, Maria Moloney, and Damien Magoni. 2018. A survey on health monitoring systems for health smart homes. International Journal of Industrial Ergonomics 66 (2018), 26–56.Google ScholarCross Ref
- Arseny Nerinovsky. 2019. Visualizations of the i3d network. https://github.com/Arseny-N/c3d-vis.Google Scholar
- Yoosuf Nizam and M Mahadi Abdul Jamil. 2020. Classification of Daily Life Activities for Human Fall Detection: A Systematic Review of the Techniques and Approaches. Studies in Systems, Decision and Control 273 (2020), 137 – 179.Google Scholar
- Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier.Google Scholar
- Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding.Google Scholar
- Karen Simonyan and Andrew Zisserman. 2014. Two-Stream Convolutional Networks for Action Recognition in Videos.Google Scholar
- Vignesh Srinivasan, Sebastian Lapuschkin, Cornelius Hellge, Klaus-Robert Müller, and Wojciech Samek. 2017. Interpretable human action recognition in compressed domain. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1692–1696.Google ScholarDigital Library
- Alexandros Stergiou, Georgios Kapidis, Grigorios Kalliatakis, Christos Chrysoulas, Remco Veltkamp, and Ronald Poppe. 2019. Saliency Tubes: Visual Explanations for Spatio-Temporal Convolutions. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE.Google Scholar
- Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2014. Learning Spatiotemporal Features with 3D Convolutional Networks.Google Scholar
- Md. Zia Uddin, Weria Khaksar, and Jim Torresen. 2018. Ambient Sensors for Elderly Care and Independent Living: A Survey. Sensors 18, 7 (2018).Google ScholarCross Ref
- Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. 2016. Temporal Segment Networks: Towards Good Practices for Deep Action Recognition.Google Scholar
- Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. 2020. Temporal Pyramid Network for Action Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
- Hong-Bo Zhang, Yi-Xiang Zhang, Bineng Zhong, Qing Lei, Lijie Yang, Ji-Xiang Du, and Duan-Sheng Chen. 2019. A Comprehensive Survey of Vision-Based Human Action Recognition Methods. Sensors 19, 5 (2019).Google Scholar
- Yinlong Zhang, Xiaoyan Zheng, Wei Liang, Sichao Zhang, and Xudong Yuan. 2022. Visual Surveillance for Human Fall Detection in Healthcare IoT. IEEE Multimedia 29 (1 2022), 36 – 46. Issue 1.Google ScholarCross Ref
Index Terms
- Explainable activity recognition for the elderly
Recommendations
An ambient assisted living nighttime wandering system for elderly
iiWAS '16: Proceedings of the 18th International Conference on Information Integration and Web-based Applications and ServicesThe Assistive living technologies provide good results for the support of specific activities transforming a home into a smart home. In this paper, we present a personalized ambient support system for elderly suffering from Alzheimer's dementia and ...
A Hierarchical Approach in Food and Drink Intake Recognition Using Wearable Inertial Sensors
PETRA '18: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments ConferenceDespite the increasing attention given to inertial sensors for Human Activity Recognition (HAR), efforts are principally focused on fitness applications where quasi-periodic activities like walking or running are studied. In contrast, activities like ...
Activity recognition with hand-worn magnetic sensors
Activity recognition is a key technology for realizing ambient assisted living applications such as care of the elderly and home automation. This paper proposes a new activity recognition method that employs hand-worn magnetic sensors to recognize a ...
Comments