Skip to main content
SpringerLink
Log in

Self-feeding frequency estimation and eating action recognition from skeletal representation using Kinect

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Under healthcare research, eating activity detection and recognition have been studied for many years. Most of the previous approaches rely on body worn sensors for eating behavior detection. However, measurement errors from these sensors will largely reduce the tracking accuracy in estimating velocity and acceleration. To avoid this problem, we utilize Microsoft Kinect to capture skeleton motions of eating and drinking behaviors. In this paper we introduce a moving average method to remove the noise in relative distances of the captured joint positions so that eating activity can be segmented into feeding and non-feeding frames. In order to identify different eating patterns, eating and drinking behavior recognition is performed based on the features extracted from the resulting feeding periods. The experiments are evaluated on our collected eating and drinking action dataset, and we also change the distance between Kinect and subject to test the robustness of our approach. The results achieve better detection and recognition performance compared with other approaches. This pioneer work of our eating action behavior analysis can lead to many potential applications such as the development of a Web system to facilitate people to share and search their eating and drinking actions as well as carrying out intelligent analysis to provide suggestions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9

Similar content being viewed by others

References

  1. Abdur Rahman, M., Qamar, A.M., Ahmed, M.A., Ataur Rahman, M., Basalamah, S.: Multimedia interactive therapy environment for children having physical disabilities. In: Proceedings of the 3rd ACM Conference on International Conference on Multimedia Retrieval, pp. 313–314 (2013)

  2. Amft, O., Bannach, D., Pirkl, G., Kreil, M., Lukowicz, P.: Towards wearable sensing-based assessment of fluid intake. In: The 8Th IEEE International Conference on Pervasive Computing and Communications Workshops, pp. 298–303 (2010)

  3. Bian, Z.P., Chau, L.P., Magnenat-Thalmann, N.: Fall detection based on skeleton extraction. In: Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry, pp. 91–94 (2012)

  4. Chang, C.Y., Lange, B., Zhang, M., Koenig, S., Requejo, P., Somboon, N., Sawchuk, A.A., Rizzo, A.A.: Towards pervasive physical rehabilitation using microsoft Kinect. In: The 6Th IEEE International Conference on Pervasive Computing Technologies for Healthcare, pp. 159–162 (2012)

  5. Chen, M., Dhingra, K., Wu, W., Yang, L., Sukthankar, R., Yang, J.: Pfid: Pittsburgh fast-food image dataset. In: The 16Th IEEE International Conference on Image Processing, pp. 289–292 (2009)

  6. Dong, Y., Hoover, A., Scisco, J., Muth, E.: A new method for measuring meal intake in humans via automated wrist motion tracking. Appl. Psychophysiol. Biofeedback 37(3), 205–215 (2012)

    Article  Google Scholar 

  7. Du, Y., Wang, W., Wang, L.: Hierarchical recurrent neural network for skeleton based action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1110–1118 (2015)

  8. Gao, C., Yang, L., Du, Y., Feng, Z., Liu, J.: From constrained to unconstrained datasets: an evaluation of local action descriptors and fusion strategies for interaction recognition. World Wide Web 19(2), 265–276 (2016)

    Article  Google Scholar 

  9. Hartigan, J.A., Wong, M.A.: Algorithm as 136: a k-means clustering algorithm. J. R. Stat. Soc.: Ser. A: Appl. Stat. 28(1), 100–108 (1979)

    MATH  Google Scholar 

  10. Hondori, H.M., Khademi, M., Lopes, C.V.: Monitoring intake gestures using sensor fusion (microsoft Kinect and inertial sensors) for smart home tele-rehab setting. In: The 1St Annual IEEE Healthcare Innovation Conference (2012)

  11. Kalantarian, H., Alshurafa, N., Sarrafzadeh, M.: A Wearable Nutrition Monitoring System The 11Th IEEE International Conference on Wearable and Implantable Body Sensor Networks, pp. 75–80 (2014)

  12. Kalantarian, H., Alshurafa, N., Le, T., Sarrafzadeh, M.: Monitoring eating habits using a piezoelectric sensor-based necklace. Comput. Biol. Med. 58, 46–55 (2015)

    Article  Google Scholar 

  13. Kawano, Y., Yanai, K.: Real-time mobile food recognition system. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–7 (2013)

  14. Li, M., Leung, H.: Graph-based approach for 3d human skeletal action recognition. Pattern Recogn. Lett. 87, 195–202 (2017)

    Article  Google Scholar 

  15. Li, M., Leung, H., Liu, Z., Zhou, L.: 3d human motion retrieval using graph kernels based on adaptive graph construction. Comput. Graph. 54, 104–112 (2016)

    Article  Google Scholar 

  16. Malnick, S.D., Knobler, H.: The medical complications of obesity. J. Assoc. Physicians 99(9), 565–579 (2006)

    Google Scholar 

  17. Mastorakis, G., Makris, D.: Fall detection system using kinect’s infrared sensor. J. Real-Time Image Proc. 9(4), 635–646 (2014)

    Article  Google Scholar 

  18. Mettel, M.R., Alekseew, M., Stocklöw, C., Braun, A.: Safety services in smart environments using depth cameras. In: European Conference on Ambient Intelligence, pp. 80–93. Springer, Switzerland (2017)

  19. Nguyen, D.T., Cohen, E., Pourhomayoun, M., Alshurafa, N.: Swallownet: Recurrent neural network detects and characterizes eating patterns. In: IEEE International Conference on Pervasive Computing and Communications Workshops, pp. 401–406 (2017)

  20. Press, W.H.: Numerical recipes 3rd edition: The art of scientific computing. Cambridge University Press, Cambridge (2007)

    MATH  Google Scholar 

  21. Qian, R.J., Sezan, M.I., Matthews, K.E.: A robust real-time face tracking algorithm. In: Proceedings of the IEEE International Conference on Image Processing, vol. 1, pp. 131–135. IEEE (1998)

  22. Saini, S., Rambli, D.R.A., Sulaiman, S., Zakaria, M.N., Shukri, S.R.M.: A low-cost game framework for a home-based stroke rehabilitation system. In: IEEE International Conference on Computer & Information Science, vol. 1, pp. 55–60 (2012)

  23. Shen, Y., Salley, J., Muth, E., Hoover, A.: Assessing the accuracy of a wrist motion tracking method for counting bites across demographic and food variables. IEEE J. Biomed. Health Inform. 21(3), 599–606 (2017)

    Article  Google Scholar 

  24. Stunkard, A., Wolff, H.: A mechanism of satiety-function and disorder in Human obesity. In: Psychosomatic Medicine, vol. 18, pp. 515–515 (1956)

  25. Tang, J.K., Leung, H.: Retrieval of logically relevant 3d human motions by adaptive feature selection with graded relevance feedback. Pattern Recogn. Lett. 33(4), 420–430 (2012)

    Article  Google Scholar 

  26. Thomaz, E., Essa, I., Abowd, G.D.: A practical approach for recognizing eating moments with wrist-mounted inertial sensing. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 1029–1040 (2015)

  27. Vemulapalli, R., Arrate, F., Chellappa, R.: Human action recognition by representing 3d skeletons as points in a lie group. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 588–595 (2014)

  28. Wu, W., Yang, J.: Fast food recognition from videos of eating for calorie estimation. In: IEEE International Conference on Multimedia and Expo, pp. 1210–1213 (2009)

  29. Yang, S., Chen, M., Pomerleau, D., Sukthankar, R.: Food recognition using statistics of pairwise local features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2249–2256 (2010)

  30. Zhang, R., Amft, O.: Monitoring chewing and eating in free-living using smart eyeglasses. IEEE J. Biomed. Health Inform. 22(1), 23–32 (2018)

    Article  Google Scholar 

  31. Zhang, S., Ang, M.H., Xiao, W., Tham, C.K.: Detection of activities by wireless sensors for daily life surveillance: eating and drinking. Sensors 9(3), 1499–1517 (2009)

    Article  Google Scholar 

  32. Zhang, Z., Liu, W., Metsis, V., Athitsos, V.: A viewpoint-independent statistical method for fall detection. In: The 21St IEEE International Conference on Pattern Recognition, pp. 3626–3630 (2012)

  33. Zong, Z., Nguyen, D.T., Ogunbona, P., Li, W.: On the combination of local texture and global structure for food classification. In: IEEE International Symposium on Multimedia, pp. 204–211 (2010)

Download references

Acknowledgements

The work described in this paper is supported by grants from City University of Hong Kong (Project No. 7004548 and 7004681), National Natural Science Foundation of China (Grant No. 61402205), China Postdoctoral Science Foundation (Grant No. 2015M571688), and Jiangsu University (Grant No. 13JDG085).

Author information

Authors and Affiliations

  1. Department of Computer Science, City University of Hong Kong, 83 Tat Chee Avenue, Kowloon, Hong Kong

    Qianhui Men & Howard Leung

  2. School of Computer Science and Communication Engineering, Jiangsu University, Jiangsu, China

    Yang Yang

Authors
  1. Qianhui Men
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Howard Leung
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qianhui Men.

Additional information

This article belongs to the Topical Collection: Special Issue on Social Media and Interactive Technologies

Guest Editors: Timothy K. Shih, Lin Hui, Somchoke Ruengittinun, and Qing Li

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Men, Q., Leung, H. & Yang, Y. Self-feeding frequency estimation and eating action recognition from skeletal representation using Kinect. World Wide Web 22, 1343–1358 (2019). https://doi.org/10.1007/s11280-018-0567-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-018-0567-0

Keywords

Search

Navigation