Skip to main content
SpringerLink
Log in

TEZARNet: TEmporal Zero-Shot Activity Recognition Network

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1969))

Included in the following conference series:

  • 364 Accesses

Abstract

Human Activity Recognition (HAR) using Inertial Measurement Unit (IMU) sensor data has practical applications in healthcare and assisted living environments. However, its use in real-world scenarios has been limited due to the lack of comprehensive IMU-based HAR datasets covering various activities. Zero-shot HAR (ZS-HAR) can overcome these data limitations. However, most existing ZS-HAR methods based on IMU data rely on attributes or word embeddings of class labels as auxiliary data to relate the seen and unseen classes. This approach requires expert knowledge and lacks motion-specific information. In contrast, videos depicting various human activities provide valuable information for ZS-HAR based on inertial sensor data, and they are readily available. Our proposed model, TEZARNet: TEmporal Zero-shot Activity Recognition Network, uses videos as auxiliary data and employs a Bidirectional Long-Short Term IMU encoder to exploit temporal information, distinguishing it from current work. The proposed model outperforms the state-of-the-art accuracy by 4.7%, 7.8%, 3.7%, and 9.3% for benchmark datasets PAMAP2, DaLiAc, UTD-MHAD, and MHEALTH, respectively. The code is available at https://github.com/nipdep/TEZARNet

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ahmed, N., Rafiq, J.I., Islam, M.R.: Enhanced human activity recognition based on smartphone sensor data using hybrid feature selection model. Sensors 20(1), 317 (2020)

    Article  Google Scholar 

  2. Banos, O., et al.: mhealthdroid: a novel framework for agile development of mobile health applications. In: Proceedings of the Ambient Assisted Living and Daily Activities: 6th International Work-Conference, pp. 91–98 (2014)

    Google Scholar 

  3. Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)

    Google Scholar 

  4. Chen, C., Jafari, R., Kehtarnavaz, N.: Utd-mhad: a multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. In: Proceedings of the IEEE International Conference on Image Processing (ICIP), pp. 168–172 (2015)

    Google Scholar 

  5. Cheng, H.T., Griss, M., Davis, P., Li, J., You, D.: Towards zero-shot learning for human activity recognition using semantic attribute sequence model. In: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 355–358 (2013)

    Google Scholar 

  6. Cheng, H.T., Sun, F.T., Griss, M., Davis, P., Li, J., You, D.: Nuactiv: recognizing unseen new activities using semantic attribute-based learning. In: Proceeding of the 11th annual international conference on Mobile Systems, Applications, and Services, pp. 361–374 (2013)

    Google Scholar 

  7. Church, K.W.: Word2vec. Nat. Lang. Eng. 23(1), 155–162 (2017)

    Article  Google Scholar 

  8. Khojasteh, S.B., Villar, J.R., Chira, C., González, V.M., De la Cal, E.: Improving fall detection using an on-wrist wearable accelerometer. Sensors 18(5), 1350 (2018)

    Article  Google Scholar 

  9. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Yann LeCun, Y.B. (ed.) Proceedings of the 3rd International Conference on Learning Representations, ICLR (2015)

    Google Scholar 

  10. Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: HMDB: a large video database for human motion recognition. In: Proceedings of the International Conference on Computer Vision (ICCV) (2011)

    Google Scholar 

  11. Leutheuser, H., Schuldhaus, D., Eskofier, B.M.: Hierarchical, multi-sensor based classification of daily life activities: comparison with state-of-the-art algorithms using a benchmark dataset. PLoS ONE 8(10), e75196 (2013)

    Article  Google Scholar 

  12. Li, Y., Wang, L.: Human activity recognition based on residual network and bilstm. Sensors 22(2), 635 (2022)

    Article  Google Scholar 

  13. Matsuki, M., Lago, P., Inoue, S.: Characterizing word embeddings for zero-shot sensor-based human activity recognition. Sensors 19(22), 5043 (2019)

    Article  Google Scholar 

  14. Meedeniya, D.: Deep Learning: A Beginners’ Guide. CRC Press LLC (2023). https://www.routledge.com/9781032473246

  15. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Proceedings of the Advances in Neural Information Processing Systems, pp. 8024–8035 (2019)

    Google Scholar 

  16. Reiss, A., Stricker, D.: Introducing a new benchmarked dataset for activity monitoring. In: Proceedings of the 16th International Symposium on Wearable Computers, pp. 108–109 (2012)

    Google Scholar 

  17. Tong, C., Ge, J., Lane, N.D.: Zero-shot learning for imu-based activity recognition using video embeddings. In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 5(4), pp. 1–23 (2021)

    Google Scholar 

  18. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)

    Google Scholar 

  19. Wang, Q., Chen, K.: Alternative semantic representations for zero-shot human action recognition. In: Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 87–102 (2017)

    Google Scholar 

  20. Wu, T., Chen, Y., Gu, Y., Wang, J., Zhang, S., Zhechen, Z.: Multi-layer cross loss model for zero-shot human activity recognition. In: Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 210–221 (2020)

    Google Scholar 

  21. Zebin, T., Sperrin, M., Peek, N., Casson, A.J.: Human activity recognition from inertial sensor time-series using batch normalized deep lstm recurrent networks. In: Proceedings of the 40th annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1–4 (2018)

    Google Scholar 

  22. Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., Eickhoff, C.: A transformer-based framework for multivariate time series representation learning. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2114–2124 (2021)

    Google Scholar 

  23. Zhang, H., Xiao, Z., Wang, J., Li, F., Szczerbicki, E.: A novel iot-perceptive human activity recognition (har) approach using multihead convolutional attention. IEEE Internet Things J. 7(2), 1072–1080 (2019)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Moratuwa, Moratuwa, Sri Lanka

    Pathirage N. Deelaka, Devin Y. De Silva, Sandareka Wickramanayake & Dulani Meedeniya

  2. School of Computing, National University of Singapore, Singapore, Singapore

    Sanka Rasnayaka

Authors
  1. Pathirage N. Deelaka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Devin Y. De Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sandareka Wickramanayake
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dulani Meedeniya
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sanka Rasnayaka
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sandareka Wickramanayake .

Editor information

Editors and Affiliations

  1. School of Automation, Central South University, Changsha, China

    Biao Luo

  2. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Long Cheng

  3. Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou, China

    Zheng-Guang Wu

  4. School of Automation, Guangdong University of Technology, Guangzhou, China

    Hongyi Li

  5. School of Electrical Engineering and Telecommunications, UNSW Sydney, Sydney, NSW, Australia

    Chaojie Li

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Deelaka, P.N., De Silva, D.Y., Wickramanayake, S., Meedeniya, D., Rasnayaka, S. (2024). TEZARNet: TEmporal Zero-Shot Activity Recognition Network. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1969. Springer, Singapore. https://doi.org/10.1007/978-981-99-8184-7_34

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8184-7_34

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8183-0

  • Online ISBN: 978-981-99-8184-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation