Skip to main content

Advertisement

SpringerLink
Log in

Towards a multimodal human activity dataset for healthcare

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Human activity recognition (HAR) based on wearable devices has become a hot topic due to the wide adoption of smartphones and smart bands. In this paper, we propose a new dataset, MMC-PCL-Activity, for wearable device-based HAR. It contains data of accelerometers, gyroscopes, heart rates, steps, GPS, weather information, mobile APP usage, and images collected from 14 participants performing 16 different types of daily activities. Besides the activity annotations, labels of physical health status and mental health status are also provided. We demonstrate the importance of multimodal fusion in activity recognition and provide baselines for more researchers using this dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://drive.google.com/file/d/1m7Qij9xqZfkDsP9TRBDqN6dI1XofUc18/view

References

  1. Abbas, A., Bilal, H.S.M., Lee, S.: Smartphone based wellness application for healthy lifestyle promotion. In: Song J., Kim M., Lane N.D., Balan R.K., Kawsar F., Liu Y. (eds.) Proceedings of the 17th annual international conference on mobile systems, applications, and services, MobiSys 2019, Seoul, Republic of Korea, June 17–21, 2019, pp. 622–623. ACM (2019). https://doi.org/10.1145/3307334.3328656

  2. Alemdar, H.Ö., Ertan, H., Incel, Ö.D., Ersoy, C.: ARAS human activity datasets in multiple homes with multiple residents. In: 7th International conference on pervasive computing technologies for healthcare and workshops, PervasiveHealth 2013, Venice, Italy, May 5–8, 2013, pp. 232–235. IEEE (2013). https://doi.org/10.4108/icst.pervasivehealth.2013.252120

  3. Anguita, D., Ghio, A., Oneto, L., Parra, X., Reyes-Ortiz, J.L.: A public domain dataset for human activity recognition using smartphones. In: 21st European symposium on artificial neural networks, ESANN 2013, Bruges, Belgium, April 24–26, 2013 (2013). http://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2013-84.pdf

  4. Bächlin, M., Plotnik, M., Roggen, D., Maidan, I., Hausdorff, J.M., Giladi, N., Tröster, G.: Wearable assistant for parkinson’s disease patients with the freezing of gait symptom. IEEE Trans. Inf. Technol. Biomed. 14(2), 436–446 (2010). https://doi.org/10.1109/TITB.2009.2036165

    Article  Google Scholar 

  5. Ballas, N., Yao, L., Pal, C., Courville, A.C.: Delving deeper into convolutional networks for learning video representations. In: Bengio Y., LeCun Y. (eds.) 4th International conference on learning representations, ICLR 2016, San Juan, Puerto Rico, May 2–4, 2016, Conference Track Proceedings (2016). arxiv: 1511.06432

  6. Baños, O., García, R., Terriza, J.A.H., Damas, M., Pomares, H., Ruiz, I.R., Saez, A., Villalonga, C.: mhealthdroid: A novel framework for agile development of mobile health applications. In: Pecchia L., Chen L.L., Nugent C.D., Bravo J. (eds.) Ambient Assisted Living and Daily Activities - 6th International Work-Conference, IWAAL 2014, Belfast, UK, December 2–5, 2014. Proceedings, Lecture Notes in Computer Science, vol. 8868, pp. 91–98. Springer (2014). https://doi.org/10.1007/978-3-319-13105-4_14

  7. Bernal, E.A., Yang, X., Li, Q., Kumar, J., Madhvanath, S., Ramesh, P., Bala, R.: Deep temporal multimodal fusion for medical procedure monitoring using wearable sensors. IEEE Trans. Multimed. 20(1), 107–118 (2018). https://doi.org/10.1109/TMM.2017.2726187

    Article  Google Scholar 

  8. Bins Filho, J.C.: Context aware vision using image-based active recognition (2004)

  9. Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S.T., Tröster, G., del R. Millán, J., Roggen, D.: The opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognit. Lett. 34(15), 2033–2042 (2013). https://doi.org/10.1016/j.patrec.2012.12.014

  10. Chen, C., Jafari, R., Kehtarnavaz, N.: UTD-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. In: 2015 IEEE International Conference on Image Processing, ICIP 2015, Quebec City, QC, Canada, September 27–30, 2015, pp. 168–172. IEEE (2015). https://doi.org/10.1109/ICIP.2015.7350781

  11. Chen, C.C., Ryoo, M., Aggarwal, J.: Ut-tower dataset: aerial view activity classification challenge (2010)

  12. Ermes, M., Pärkkä, J., Mäntyjärvi, J., Korhonen, I.: Detection of daily activities and sports with wearable sensors in controlled and uncontrolled conditions. IEEE Trans. Inf. Technol. Biomed. 12(1), 20–26 (2008). https://doi.org/10.1109/TITB.2007.899496

    Article  Google Scholar 

  13. Fisher, R.: Behave: Computer-assisted prescreening of video streams for unusual activities. The EPSRC project GR S 98146 (2007)

  14. Gorelick, L., Blank, M., Shechtman, E., Irani, M., Basri, R.: Actions as space-time shapes. Trans. Pattern Anal. Mach. Intell. 29(12), 2247–2253 (2007)

    Article  Google Scholar 

  15. Hammerla, N.Y., Halloran, S., Plötz, T.: Deep, convolutional, and recurrent models for human activity recognition using wearables. In: S. Kambhampati (ed.) Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pp. 1533–1540. IJCAI/AAAI Press (2016). http://www.ijcai.org/Abstract/16/220

  16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). arxiv: 1512.03385

  17. Hsieh, P., Lin, Y., Chen, Y., Hsu, W.H.: Egocentric activity recognition by leveraging multiple mid-level representations. In: IEEE international conference on multimedia and expo, ICME 2016, Seattle, WA, USA, July 11-15, 2016, pp. 1–6. IEEE Computer Society (2016). https://doi.org/10.1109/ICME.2016.7552937

  18. Inman, V.T., Eberhart, H.D., et al.: The major determinants in normal and pathological gait. JBJS 35(3), 543–558 (1953)

    Article  Google Scholar 

  19. Ji, S., Xu, W., Yang, M., Yu, K.: 3d convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221–231 (2013). https://doi.org/10.1109/TPAMI.2012.59

    Article  Google Scholar 

  20. van Kasteren, T.L., Englebienne, G., Kröse, B.J.: Human activity recognition from wireless sensor network data: Benchmark and software. In: Activity recognition in pervasive intelligent environments, pp. 165–186. Springer (2011)

  21. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Y. Bengio, Y. LeCun (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings (2015). arxiv: 1412.6980

  22. Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: HMDB: a large video database for human motion recognition. In: Proceedings of the international conference on computer vision (ICCV) (2011)

  23. Kwapisz, J.R., Weiss, G.M., Moore, S.: Activity recognition using cell phone accelerometers. SIGKDD Explor. 12(2), 74–82 (2010). https://doi.org/10.1145/1964897.1964918

    Article  Google Scholar 

  24. Lee, M.L., Dey, A.K.: Sensor-based observations of daily living for aging in place. Pers. Ubiquitous Comput. 19(1), 27–43 (2015). https://doi.org/10.1007/s00779-014-0810-3

    Article  Google Scholar 

  25. de Lourdes Martínez-Villaseñor, M., Ponce, H., Brieva, J., Moya-Albor, E., Núñez-Martínez, J., Peñafort-Asturiano, C.: Up-fall detection dataset: a multimodal approach. Sensors 19(9), 1988 (2019). https://doi.org/10.3390/s19091988

    Article  Google Scholar 

  26. Majumder, S., Kehtarnavaz, N.: Vision and inertial sensing fusion for human action recognition : A review. CoRR abs/2008.00380 (2020). arxiv: 2008.00380

  27. Malekzadeh, M., Clegg, R.G., Cavallaro, A., Haddadi, H.: Mobile sensor data anonymization. In: O. Landsiedel, K. Nahrstedt (eds.) Proceedings of the International Conference on Internet of Things Design and Implementation, IoTDI 2019, Montreal, QC, Canada, April 15–18, 2019, pp. 49–58. ACM (2019). https://doi.org/10.1145/3302505.3310068

  28. Marszalek, M., Laptev, I., Schmid, C.: Actions in context. In: 2009 IEEE computer society conference on computer vision and pattern recognition (CVPR 2009), 20–25 June 2009, Miami, Florida, USA, pp. 2929–2936. IEEE Computer Society (2009). https://doi.org/10.1109/CVPR.2009.5206557

  29. Martín, D.R., Samà, A., Pérez-López, C., Català, A., Cabestany, J., Rodríguez-Molinero, A.: Svm-based posture identification with a single waist-located triaxial accelerometer. Expert Syst. Appl. 40(18), 7203–7211 (2013). https://doi.org/10.1016/j.eswa.2013.07.028

    Article  Google Scholar 

  30. Mauldin, T.R., Canby, M.E., Metsis, V., Ngu, A.H.H., Rivera, C.C.: Smartfall: a smartwatch-based fall detection system using deep learning. Sensors 18(10), 3363 (2018). https://doi.org/10.3390/s18103363

    Article  Google Scholar 

  31. Morris, J.: Accelerometry-a technique for the measurement of human body movements. J Biomech 6(6), 729–736 (1973)

    Article  Google Scholar 

  32. Nakamura, K., Yeung, S., Alahi, A., Fei-Fei, L.: Jointly learning energy expenditures and activities using egocentric multimodal signals. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 6817–6826. IEEE Computer Society (2017). https://doi.org/10.1109/CVPR.2017.721

  33. Newman, S.: building microservices-designing fine-grained systems, 1st Edition. O’Reilly (2015). https://www.worldcat.org/oclc/904463848

  34. Noor, M.H.M., Salcic, Z.A., Wang, K.I.: Adaptive sliding window segmentation for physical activity recognition using a single tri-axial accelerometer. Pervasive Mob. Comput. 38, 41–59 (2017). https://doi.org/10.1016/j.pmcj.2016.09.009

    Article  Google Scholar 

  35. Oh, H., Jain, R.C.: From multimedia logs to personal chronicles. In: Q. Liu, R. Lienhart, H. Wang, S.K. Chen, S. Boll, Y.P. Chen, G. Friedland, J. Li, S. Yan (eds.) Proceedings of the 2017 ACM on Multimedia Conference, MM 2017, Mountain View, CA, USA, October 23–27, 2017, pp. 881–889. ACM (2017). https://doi.org/10.1145/3123266.3123375

  36. Pham, C., Olivier, P.: Slice&dice: Recognizing food preparation activities using embedded accelerometers. In: M. Tscheligi, B.E.R. de Ruyter, P. Markopoulos, R. Wichert, T. Mirlacher, A. Meschtscherjakov, W. Reitberger (eds.) Ambient Intelligence, European Conference, AmI 2009, Salzburg, Austria, November 18-21, 2009. Proceedings, Lecture Notes in Computer Science, vol. 5859, pp. 34–43. Springer (2009). https://doi.org/10.1007/978-3-642-05408-2_4

  37. Reiss, A., Stricker, D.: Introducing a new benchmarked dataset for activity monitoring. In: 16th International symposium on wearable Computers, ISWC 2012, Newcastle, United Kingdom, June 18–22, 2012, pp. 108–109. IEEE Computer Society (2012). https://doi.org/10.1109/ISWC.2012.13

  38. Schüldt, C., Laptev, I., Caputo, B.: Recognizing human actions: A local SVM approach. In: 17th International conference on pattern recognition, ICPR 2004, Cambridge, UK, August 23–26, 2004, pp. 32–36. IEEE Computer Society (2004). https://doi.org/10.1109/ICPR.2004.1334462

  39. Shoaib, M., Bosch, S., Incel, Ö.D., Scholten, H., Havinga, P.J.M.: Fusion of smartphone motion sensors for physical activity recognition. Sensors 14(6), 10146–10176 (2014). https://doi.org/10.3390/s140610146

    Article  Google Scholar 

  40. Singla, G., Cook, D.J., Schmitter-Edgecombe, M.: Recognizing independent and joint activities among multiple residents in smart environments. J. Ambient Intell. Humaniz. Comput. 1(1), 57–63 (2010). https://doi.org/10.1007/s12652-009-0007-1

    Article  Google Scholar 

  41. Song, S., Chandrasekhar, V., Mandal, B., Li, L., Lim, J., Babu, G.S., San, P.P., Cheung, N.: Multimodal multi-stream deep learning for egocentric activity recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2016, Las Vegas, NV, USA, June 26–July 1, 2016, pp. 378–385. IEEE Computer Society (2016). https://doi.org/10.1109/CVPRW.2016.54

  42. Soraya, S.I., Chuang, S., Tseng, Y., Ik, T., Ching, Y.: A comprehensive multisensor dataset employing RGBD camera, inertial sensor and web camera. In: 20th Asia-Pacific Network Operations and Management Symposium, APNOMS 2019, Matsue, Japan, September 18–20, 2019, pp. 1–4. IEEE (2019). https://doi.org/10.23919/APNOMS.2019.8892906

  43. Spriggs, E.H., la Torre, F.D., Hebert, M.: Temporal segmentation and activity classification from first-person sensing. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2009, Miami, FL, USA, 20-25 June, 2009, pp. 17–24. IEEE Computer Society (2009). https://doi.org/10.1109/CVPRW.2009.5204354

  44. Stisen, A., Blunck, H., Bhattacharya, S., Prentow, T.S., Kjærgaard, M.B., Dey, A.K., Sonne, T., Jensen, M.M.: Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In: Song J., Abdelzaher T.F., Mascolo C. (eds.) Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems, SenSys 2015, Seoul, South Korea, November 1–4, 2015, pp. 127–140. ACM (2015). https://doi.org/10.1145/2809695.2809718

  45. Sun, L., Aizawa, K.: Action recognition using invariant features under unexampled viewing conditions. In: Jaimes A., Sebe N., Boujemaa N., Gatica-Perez D., Shamma D.A., Worring M., Zimmermann R. (eds.) ACM Multimedia Conference, MM ’13, Barcelona, Spain, October 21–25, 2013, pp. 389–392. ACM (2013). https://doi.org/10.1145/2502081.2508126

  46. Tapia, E.M., Intille, S.S., Haskell, W.L., Larson, K., Wright, J.A., King, A., Friedman, R.H.: Real-time recognition of physical activities and their intensities using wireless accelerometers and a heart rate monitor. In: 11th IEEE International Symposium on Wearable Computers (ISWC 2007), October 11–13, 2007, Boston, MA, USA, pp. 37–40. IEEE Computer Society (2007). https://doi.org/10.1109/ISWC.2007.4373774

  47. Um, T.T., Pfister, F.M.J., Pichler, D., Endo, S., Lang, M., Hirche, S., Fietzek, U., Kulic, D.: Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks. In: Lank E., Vinciarelli A., Hoggan E.E., Subramanian S., Brewster S.A. (eds.) Proceedings of the 19th ACM International Conference on Multimodal Interaction, ICMI 2017, Glasgow, United Kingdom, November 13–17, 2017, pp. 216–220. ACM (2017). https://doi.org/10.1145/3136755.3136817

  48. Vavoulas, G., Chatzaki, C., Malliotakis, T., Pediaditis, M., Tsiknakis, M.: The mobiact dataset: Recognition of activities of daily living using smartphones. In: Röcker C., Ziefle M., O’Donoghue J., Maciaszek L.A., Molloy W. (eds.) Proceedings of the 2nd International Conference on Information and Communication Technologies for Ageing Well and e-Health, ICT4AgeingWell 2016, Rome, Italy, April 21–22, 2016, pp. 143–151. SCITEPRESS (2016). https://doi.org/10.5220/0005792401430151

  49. Vu, T., Olsson, C., Laptev, I., Oliva, A., Sivic, J.: Predicting actions from static scenes. In: Fleet D.J., Pajdla T., Schiele B., Tuytelaars T. (eds.) Computer Vision-ECCV 2014-13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part V, Lecture Notes in Computer Science, vol. 8693, pp. 421–436. Springer (2014). https://doi.org/10.1007/978-3-319-10602-1_28

  50. Wang, N., Ambikairajah, E., Lovell, N.H., Celler, B.G.: Accelerometry based classification of walking patterns using time-frequency analysis. In: 2007 29th annual international conference of the ieee engineering in medicine and biology society, pp. 4899–4902. IEEE (2007)

  51. Weinland, D., Ronfard, R., Boyer, E.: Free viewpoint action recognition using motion history volumes. Comput. Vis. Image Underst. 104(2–3), 249–257 (2006). https://doi.org/10.1016/j.cviu.2006.07.013

    Article  Google Scholar 

  52. Yao, B., Jiang, X., Khosla, A., Lin, A.L., Guibas, L.J., Li, F.: Human action recognition by learning bases of action attributes and parts. In: Metaxas D.N., Quan L., Sanfeliu A., Gool L.V. (eds.) IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6–13 , pp. 1331–1338. (2011) IEEE Computer Society (2011). https://doi.org/10.1109/ICCV.2011.6126386

  53. Yao, B., Li, F.: Grouplet: A structured image representation for recognizing human and object interactions. In: The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pp. 9–16. IEEE Computer Society (2010). https://doi.org/10.1109/CVPR.2010.5540234

  54. Zhang, M., Sawchuk, A.A.: USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors. In: A.K. Dey, H. Chu, G.R. Hayes (eds.) The 2012 ACM Conference on Ubiquitous Computing, Ubicomp ’12, Pittsburgh, PA, USA, September 5–8, 2012, pp. 1036–1043. ACM (2012). https://doi.org/10.1145/2370216.2370438

Download references

Acknowledgements

This work was supported by National Key Research and Development Program of China (No. 2018AAA0100604), National Natural Science Foundation of China (No. 61720106006, 62072455, 61721004, U1836220, U1705262, 61872424).

Author information

Author notes

  1. Menghao Hu and Mingxuan Luo equally contributed.

Authors and Affiliations

  1. ZhengZhou University, ZhengZhou, 450002, China

    Menghao Hu, Mingxuan Luo, Wenhua Meng & Baochen Xiong

  2. North China University of Science and Technology, TangShan, 063210, China

    Menghua Huang

  3. National Lab of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Xiaoshan Yang

  4. Peng Cheng Laboratory, Shenzhen, China

    Xiaoshan Yang & Jitao Sang

  5. Beijing Jiaotong University, Beijing, China

    Jitao Sang

Authors
  1. Menghao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingxuan Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Menghua Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenhua Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Baochen Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaoshan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jitao Sang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoshan Yang.

Additional information

Communicated by Bing-Kun Bao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, M., Luo, M., Huang, M. et al. Towards a multimodal human activity dataset for healthcare. Multimedia Systems 29, 1–13 (2023). https://doi.org/10.1007/s00530-021-00875-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-021-00875-6

Keywords

Search

Navigation