Skip to main content
Springer Nature Link
Log in

HumanSense: a framework for collective human activity identification using heterogeneous sensor grid in multi-inhabitant smart environments

  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

A Correction to this article was published on 06 August 2020

This article has been updated

Abstract

Identification of human activity considering social interactions and group dynamics non-intrusively has been one of the fundamental problems and a challenging area of research. In real life, it is required for designing human-centric applications like assisted living, health care, and creating a smart home environment. As human beings spend 90% of time indoors, such a system will be helpful to monitor the behavioral anomalies of the inhabitants. Existing approaches have used intrusive or invasive methods like camera or wearable devices. In this work, we present a device-free, non-invasive, and non-intrusive sensing framework called HumanSense using an array of heterogeneous sensor grid for human activity monitoring. The sensor grids, comprising the ultrasonic and sound sensors, have been deployed for collective sensing combining a person’s physical activity and verbal interaction information. The proposed system senses a stream of events when the occupant(s) perform different physical activities categorized as atomic and group activities like sitting, standing, and walking. Simultaneously, it also tracks person-person verbal interactions such as monologue and discussion. Both information are then integrated into a single framework to understand the overall behavioral scenario of the indoor environment. The experimental results have shown that HumanSense can detect different activities with accuracy more than 90% and also improves overall identification accuracy compared to existing works. Our developed system can be further evolved into ready-to-deploy smart sensing panels which can be effective for human activity monitoring in an indoor environment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Change history

Notes

  1. https://www.mecoinst.com/meco-product-details/digital-sound-level-meter.aspx

References

  1. Cook DJ, Das SK (2007) How smart are our environments? an updated look at the state of the art. Pervasive Mob Comput 3(2):53–73

    Article  Google Scholar 

  2. Wilson C, Hargreaves T, Hauxwell-Baldwin R (2015) Smart homes and their users: a systematic analysis and key challenges. Pers Ubiquit Comput 19(2):463–476

    Article  Google Scholar 

  3. Lane ND, Mohammod M, Mu L, Yang X, Lu H, Ali S, Doryab A, Berke E, Choudhury T, Campbell A (2011) Bewell: a smartphone application to monitor, model and promote wellbeing

  4. Shoaib Muhammad, Bosch Stephan, Scholten Hans, Havinga Paul JM (2015) Ozlem Durmaz Incel. Towards detection of bad habits by fusing smartphone and smartwatch sensors. In: 2015 IEEE international conference on pervasive computing and communication workshops (PerCom Workshops), pp 591–596. IEEE

  5. Tran DN, Phan DD (2016) Human activities recognition in android smartphone using support vector machine. In: 2016 7th international conference on intelligent systems, modelling and simulation (ISMS), pp 64–68. IEEE

  6. Ibrahim MS, Muralidharan S, Deng Z, Vahdat A, Mori G (2016) A hierarchical deep temporal model for group activity recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 1971–1980. IEEE

  7. Ryoo MS, Aggarwal JK (2008) Recognition of high-level group activities based on activities of individual members. In: 2008 IEEE workshop on motion and video computing, 2008. WMVC, pages 1–8. IEEE

  8. Cho N-G, Kim Y-J, Park U, Park J-S, Lee S-W (2015) Group activity recognition with group interaction zone based on relative distance between human objects. Int J Pattern Recogn Artif Intell 29(05):1555007

    Article  MathSciNet  Google Scholar 

  9. Zhang D, Gatica-Perez D, Bengio S, McCowan I (2006) Modeling individual and group actions in meetings with layered hmms. IEEE Trans Multimed 8(3):509–520

    Article  Google Scholar 

  10. Yun Jaeseok, Song Min-Hwan (2014) Detecting direction of movement using pyroelectric infrared sensors. IEEE Sensors J 14(5):1482–1489

    Article  Google Scholar 

  11. Wang W, Liu AX, Shahzad M, Ling K, Lu S (2015) Understanding and modeling of wifi signal based human activity recognition. In: Proceedings of the 21st annual international conference on mobile computing and networking, pp 65–76. ACM

  12. Noceti N, Odone F (2014) Humans in groups: the importance of contextual information for understanding collective activities. Pattern Recogn 47(11):3535–3551

    Article  Google Scholar 

  13. Choi W, Savarese S (2013) Understanding collective activities of people from videos. IEEE Trans Pattern Anal Mach Intell 36(6):1242–1257

    Article  Google Scholar 

  14. Chang X, Zheng W-S, Zhang J (2015) Learning person–person interaction in collective activity recognition. IEEE Trans Image Process 24(6):1905–1918

    Article  MathSciNet  Google Scholar 

  15. Liu J, Wang C, Gong Y, Xue H (2019) Deep fully connected model for collective activity recognition, vol 7

  16. Zhang P, Tang Y, Hu J-F, Zheng W-S (2019) Fast collective activity recognition under weak supervision. IEEE Trans Image Process 29:29–43

    Article  MathSciNet  Google Scholar 

  17. Klepeis NE, Nelson WC, Ott WR, Robinson JP, Tsang AM, Switzer P, Behar JV, Hern SC, Engelmann WH (2001) The national human activity pattern survey (nhaps): a resource for assessing exposure to environmental pollutants. J Expo Sci Environ Epid 11(3):231

    Article  Google Scholar 

  18. Ugolotti R, Sassi F, Mordonini M, Cagnoni S (2013) Multi-sensor system for detection and classification of human activities. J Ambient Intell Humaniz Comput 4(1):27–41

    Article  Google Scholar 

  19. San-Segundo R, Lorenzo-Trueba J, Martínez-González B, Pardo JM (2016) Segmenting human activities based on hmms using smartphone inertial sensors. Pervasive Mob Comput 30:84–96

    Article  Google Scholar 

  20. Attal F, Mohammed S, Dedabrishvili M, Chamroukhi F, Oukhellou L, Amirat Y (2015) Physical human activity recognition using wearable sensors. Sensors 15(12):31314–31338

    Article  Google Scholar 

  21. Hardegger M, Roggen D, Tröster G (2015) 3d actionslam: wearable person tracking in multi-floor environments. Pers Ubiquit Comput 19(1):123–141

    Article  Google Scholar 

  22. Hussain S, Schaffner S, Moseychuck D (2009) Applications of wireless sensor networks and rfid in a smart home environment. In: 2009 7th annual communication networks and services research conference CNSR’09, pp 153–157. IEEE

  23. Yu Z, Zhang D, Wang Z, Guo B, Roussaki I, Doolin K, Claffey E (2017) Toward context-aware mobile social networks. IEEE Commun Mag 55(10):168–175

    Article  Google Scholar 

  24. Zhang Z, Poslad S (2014) Improved use of foot force sensors and mobile phone gps for mobility activity recognition. IEEE Sensors J 14(12):4340–4347

    Article  Google Scholar 

  25. Jang Y, Shin S, Won Lee J, Kim S (2007) A preliminary study for portable walking distance measurement system using ultrasonic sensors. In: 29th annual international conference of the IEEE engineering in medicine and biology society 2007. EMBS 2007, pp 5290–5293. IEEE

  26. Li N, Zhang N, Das SK, Thuraisingham B (2009) Privacy preservation in wireless sensor networks: A state-of-the-art survey. Ad Hoc Netw 7(8):1501–1514

    Article  Google Scholar 

  27. Al Ameen M, Liu J, Kwak K (2012) Security and privacy issues in wireless sensor networks for healthcare applications. J Med Syst 36(1):93–101

    Article  Google Scholar 

  28. Gaglio S, Lo Re G, Morana M (2015) Human activity recognition process using 3-d posture data. IEEE Trans Human-Mach Syst 45(5):586–597

    Article  Google Scholar 

  29. Mettel MR, Alekseew M, Stockl"ow C, Braun A (2018) Designing and evaluating safety services using depth cameras. J Ambient Intell Humaniz Comput 2018:1–13

    Google Scholar 

  30. Noori FM, Wallace B, Uddin Z Md, Torresen J (2019) A robust human activity recognition approach using openpose, motion features, and deep recurrent neural network. In: Scandinavian conference on image analysis, pp 299–310. Springer

  31. Cheng R, Heinzelman W, Sturge-Apple M, Ignjatovic Z (2012) A motion-tracking ultrasonic sensor array for behavioral monitoring. IEEE Sensors J 12(3):707–712

    Article  Google Scholar 

  32. De D, Tang S, Song W-Z, Cook D, Das SK (2012) Actisen: activity-aware sensor network in smart environments. Pervasive Mob Comput 8(5):730–750

    Article  Google Scholar 

  33. Alvarez-Alvarez A, Alonso JM, Trivino G (2013) Human activity recognition in indoor environments by means of fusing information extracted from intensity of wifi signal and accelerations. Inform Sci 233:162–182

    Article  Google Scholar 

  34. Zhang F, Niu K, Xiong J, Jin B, Gu T, Jiang Y, Zhang D (2019) Towards a diffraction-based sensing approach on human activity recognition. Proceedings of the ACM on Interactive Mobile, Wearable and Ubiquitous Technologies 3(1):1–25

    Google Scholar 

  35. Kim Y-J, Cho N-G, Lee S-W (2014) Group activity recognition with group interaction zone. In: 2014 22nd international conference on pattern recognition, pp 3517–3521. IEEE

  36. Stephens K, Bors AG (2018) Modelling of interactions for the recognition of activities in groups of people. Digital Signal Process 79:34–46

    Article  MathSciNet  Google Scholar 

  37. Das S, Chatterjee S, Chakraborty S, Mitra Bivas (2018) Meetsense: a lightweight framework for group identification using smartphones CoRR

  38. Casagranda P, Sapino ML, Candan KS (2015) Audio assisted group detection using smartphones. In: 2015 IEEE international conference on multimedia & expo workshops (ICMEW), pp 1–6. IEEE

  39. Lu H, Pan W, Lane ND, Choudhury T, Campbell AT (2009) Soundsense: scalable sound sensing for people-centric applications on mobile phones. In: Proceedings of the 7th international conference on Mobile systems, applications, and services, pp 165–178. ACM

  40. Dai P, Di H, Dong L, Tao L, Xu G (2008) Group interaction analysis in dynamic context. IEEE Trans Syst Man Cybern Part B (Cybern) 38(1):275–282

    Article  Google Scholar 

  41. Yu C, Hansen JHL (2017) Active learning based constrained clustering for speaker diarization. IEEE/ACM Trans on Audio, Speech, and Language Process 25(11):2188–2198

    Article  Google Scholar 

  42. Rossi M, Feese S, Amft O, Braune N, Martis S, Tröster G (2013) Ambientsense: a real-time ambient sound recognition system for smartphones. In 2013 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), pp 230–235. IEEE

  43. Wang C, Xu Y, Liang H, Huang W, Zhang L (2018) Woody: a post-process method for smartphone-based activity recognition. IEEE Access 6:49611–49625

    Article  Google Scholar 

  44. Yacchirema D, de Puga JS, Palau C, Esteve M (2019) Fall detection system for elderly people using iot and ensemble machine learning algorithm. Pers Ubiquit Comput 2019:1–17

    Google Scholar 

  45. Deng Z, Zhai M, Chen L, Liu Y, Muralidharan S, Roshtkhari MJ, Mori G (2015) Deep structured models for group activity recognition. arXiv:1506.04191

  46. Kaneko T, Shimosaka M, Odashima S, Fukui R, Sato T (2014) A fully connected model for consistent collective activity recognition in videos. Pattern Recogn Lett 43:109–118

    Article  Google Scholar 

  47. Tran SN, Zhang Q (2020) Towards multi-resident activity monitoring with smarter safer home platform. In Smart assisted living, pp 249–267. Springer

  48. Siddiqi MH, Alruwaili M, Ali A (2019) A novel feature selection method for video-based human activity recognition systems, vol 7

  49. Srinivasan V, Stankovic J, Whitehouse K (2010) Using height sensors for biometric identification in multi-resident homes. In: International conference on pervasive computing, pp 337–354. Springer

  50. Ghosh A, Sanyal A, Chakraborty A, Sharma Praveen K, Saha M, Nandi S, Saha S (2017) On automatizing recognition of multiple human activities using ultrasonic sensor grid. In: 2017 9th international conference on communication systems and networks (COMSNETS), pp 488–491

  51. Nadee C, Chamnongthai K (2015) Ultrasonic array sensors for monitoring of human fall detection. In: 2015 12th international conference on electrical engineering/electronics, computer, telecommunications and information technology (ECTI-CON), pp 1–4. IEEE

  52. Liu C-D, Chung Y-N, Chung P-C (2010) An interaction-embedded hmm framework for human behavior understanding: with nursing environments as examples. IEEE Trans Inf Technol Biomed 14(5):1236–1246

    Article  MathSciNet  Google Scholar 

  53. Deng Z, Vahdat A, Hu H, Mori G (2016) Structure inference machines: recurrent neural networks for analyzing relations in group activity recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4772–4781

  54. Ghosh A, Chakraborty D, Prasad D, Saha M, Saha S (2018) Can we recognize multiple human group activities using ultrasonic sensors? In 2018 10th International Conference on Communication Systems & Networks (COMSNETS), pp 557–560. IEEE

  55. Dupont S, Luettin J (2000) Audio-visual speech modeling for continuous speech recognition. IEEE Trans Multimed 2(3):141–151

    Article  Google Scholar 

  56. Moon TK (1996) The expectation-maximization algorithm. IEEE Signal Process Mag 13(6):47–60

    Article  Google Scholar 

  57. David Forney G (1973) The viterbi algorithm. Proc IEEE 61(3):268–278

    Article  MathSciNet  Google Scholar 

  58. Han W, Chan C-F, Choy C-S, Pun K-P (2006) An efficient mfcc extraction method in speech recognition. In: 2006 IEEE international symposium on circuits and systems, pp 4–pp. IEEE

  59. Tranter SE, Reynolds DA (2006) An overview of automatic speaker diarization systems. IEEE Trans Audio, Speech, Language Process 14(5):1557–1565

    Article  Google Scholar 

  60. Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL (2012) Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. In: International workshop on ambient assisted living, pp 216–223. Springer

  61. Kim S-W, Lee Y-J, Cho N-G (2014) Group activity recognition with group interaction zone. In: 2014 22nd international conference on pattern recognition (ICPR), pp 3517–3521

  62. Heldner M, Edlund J (2010) Pauses, gaps and overlaps in conversations. J Phon 38(4):555–568

    Article  Google Scholar 

  63. Yi Z, Kuroda T (2014) Wearable sensor-based human activity recognition from environmental background sounds. J Ambient Intell Humaniz Comput 5(1):77–89

    Article  Google Scholar 

  64. Vafeiadis A, Votis K, Giakoumis D, Tzovaras D, Chen L, Hamzaoui R (2020) Audio content analysis for unobtrusive event detection in smart homes. Eng Appl Artif Intel 89:103226

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dr B C Roy Engineering College, Durgapur, India

    Arindam Ghosh

  2. Jadavpur University, Kolkata, India

    Amartya Chakraborty

  3. Tata Consultany Services Ltd., Chennai, India

    Joydeep Kumbhakar

  4. National Institute of Technology Durgapur, Durgapur, India

    Mousumi Saha & Sujoy Saha

Authors
  1. Arindam Ghosh
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Amartya Chakraborty
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Joydeep Kumbhakar
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Mousumi Saha
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Sujoy Saha
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Arindam Ghosh.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghosh, A., Chakraborty, A., Kumbhakar, J. et al. HumanSense: a framework for collective human activity identification using heterogeneous sensor grid in multi-inhabitant smart environments. Pers Ubiquit Comput 26, 521–540 (2022). https://doi.org/10.1007/s00779-020-01402-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-020-01402-6

Keywords