Skip to main content
SpringerLink
Log in

Depth-Based In-Bed Human Pose Estimation with Synthetic Dataset Generation and Deep Keypoint Estimation

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 Workshops (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13806))

Included in the following conference series:

Abstract

This paper describes a method of estimating the pose of a human in bed only from a single depth image. Such estimation is useful for robotic monitoring of the elderly and the disabled, where their lying posture may indicate illness. While it can address privacy and illumination issues, depth images make the pose estimation problem more challenging. We solve this problem by generating training images with cloth simulation and deep keypoint estimation. We evaluated the effectiveness of the dataset using synthetic and real test images. We also show that adding a small number of real training data improves the results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Maya. http://www.autodesk.com/products/maya/overview/

  2. Maya ncloth. http://knowledge.autodesk.com/support/maya/learn-explore/caas/CloudHelp/cloudhelp/2018/ENU/Maya-CharEffEnvBuild/files/GUID-ED791F1C-8412-4785-829F-9925F2604E8A-htm.html

  3. Nation sleep foundation. http://sleepfoundation.org

  4. Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: new benchmark and state of the art analysis. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)

    Google Scholar 

  5. Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (2017)

    Google Scholar 

  6. Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., Yuille, A.: Detect what you can: Detecting and representing objects using holistic models and body parts. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)

    Google Scholar 

  7. Clever, H., Erickson, Z., Kapusta, A., Turk, G., Liu, C., Kemp, C.: Bodies at rest: 3d human pose and shape estimation from a pressure image using synthetic data. In: Proceedings of 2020 IEEE Conference on Computer Vision and Pattern Recognition (2020)

    Google Scholar 

  8. Davoodnia, V., Etemad, A.: Identity and posture recognition in smart beds with deep multitask learning. In: Proceedings of 2019 IEEE International Conference on Systems, Man, and Cybernetics (2019)

    Google Scholar 

  9. Harada, T., Mori, T., Nishida, Y., Yoshimi, T., Sato, T.: Body parts positions and posture estimation system based on pressure distribution image. In: Proceedings of 1999 IEEE International Conference on Robotics and Automation (1999)

    Google Scholar 

  10. Harada, T., Sato, T., Mori, T.: Pressure distribution image based human motion tracking system using skeleton and surface integration model. In: Proceedings of 2001 IEEE International Conference on Robotics and Automation (2001)

    Google Scholar 

  11. Hasegawa, M., Hayashi, K., Miura, J.: Fatigue estimation using facial expression features and remote-PPG signal. In: Proceedings of 2019 IEEE International Conference on Robot and Human Interactive Communication (2019)

    Google Scholar 

  12. Huang, R.Y., Dung, L.R.: Measurement of heart rate variability using off-the-shelf smart phones. Biomed. Eng. Online 15(1), 1–16 (2016)

    Article  Google Scholar 

  13. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NIPS) (2012)

    Google Scholar 

  14. Liu, S., Ostadabbas, S.: A vision-based system for in-bed posture tracking. In: Proceedings of the 5th International Workshop on Assistive Computer Vision and Robotics (2017)

    Google Scholar 

  15. Liu, S., Yin, Y., Ostadabbas, S.: In-bed pose estimation: deep learning with shallow dataset. IEEE J. Transl. Eng. Health Med. 7, 1–2 (2019)

    Article  Google Scholar 

  16. Liu, Z., Zhu, J., Bu, J., Chen, C.: A survey of human pose estimation: the body parts parsing based methods. J. Visual Commun. Image Represent. 32, 10–19 (2015)

    Article  Google Scholar 

  17. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  18. Mikic, I., Huang, K., Trivedi, M.: Activity monitoring and summarization for an intelligent meeting room. In: Proceedings of IEEE Workshop on Human Motion (2000)

    Google Scholar 

  19. Moeslund, T., Hilton, A., Krüger, V.: A survey of advances in vision-based human motion capture and analysis. Comput. Vision Image Underst. 104, 90–126 (2006)

    Article  Google Scholar 

  20. Mori, T., Tominaga, S., Noguchi, H., Shimoasaka, M., Fukui, R., Sato, T.: Behavior prediction from trajectories in a house by estimating transition model using stay points. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3419–3425 (2011)

    Google Scholar 

  21. Nishi, K., Demura, M., Miura, J., Oishi, S.: Use of thermal point cloud for thermal comfort measurement and human pose estimation in robotic monitoring. In: Proceedings of 5th International Workshop on Assistive Computer Vision and Robotics (2017)

    Google Scholar 

  22. Nishi, K., Miura, J.: Generation of human depth images with body part labels for complex human pose recognition. Pattern Recogn. 71, 402–413 (2017)

    Article  Google Scholar 

  23. Ochi, S., Miura, J.: Human pose recognition uder cloth-like objects from depth images using a synthetic image dataset with cloth simulation. In: Proceedings of 2021 IEEE/SICE Int. Symposium on System Integration (2021)

    Google Scholar 

  24. Oliveira, G., Valada, A., Bollen, C., Burgard, W., Brox, T.: Deep learning for human part discovery in images. In: Proceedings of 2016 IEEE International Conference on Robotics and Automation (2016)

    Google Scholar 

  25. Pantelopoulos, A., Bourbakis, N.: A survey on wearable sensor-based systems for health monitoring and prognosis. IEEE Trans. Syst. Man, Cybern. Part C. Appl. Rev. 40(1), 1–12 (2010)

    Google Scholar 

  26. Sapp, B., Taskar, B.: Modec: multimodal decomposable models for human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2013)

    Google Scholar 

  27. Shinzaki, M., Iwashita, Y., Kurazume, R., Ogawara, K.: Gait-based person identification method using shaodow biometrics for robustness to changes in the walking direction. In: Proceedings of 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 670–677 (2015)

    Google Scholar 

  28. Shotton, J., et al.: Real-time human pose recognition in parts from single depth images. Commun. ACM 56(1), 116–124 (2013)

    Article  Google Scholar 

  29. Sun, Q., Gonzalez, E., Sun, Y.: On bed posture recognition with pressure sensor array system. In: 2016 IEEE SENSORS (2016)

    Google Scholar 

  30. Toshev, Z., Szegedy, C.: Deeppose: human pose estimation via deep neural networks. In: Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1653–1660 (2014)

    Google Scholar 

  31. Vasileiadis, M., Malassiotis, S., Giakoumis, D., Bouganis, C.S., Tzovaras, D.: Robust human pose tracking for realistic service robot applications. In: Proceedings of the 5th International Workshop on Assistive Computer Vision and Robotics (2017)

    Google Scholar 

  32. Xu, X., Lin, F., Wang, A., Song, C., Hu, Y., Xu, W.: On-bed sleep posture recognition based on body-earth mover’s distance. In: Proceedings of 2015 Biomedical Circuits and Systems Conference (2015)

    Google Scholar 

  33. Yamamoto, T., Terada, K., Ochiai, A., Saito, F., Asahara, A., Murase, K.: Development of human support robot as the research platform of a domestic mobile manipulator. ROBOMECH J. 6(1), 1–5 (2019)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Toyohashi University of Technology, Toyohashi, Japan

    Shunsuke Ochi & Jun Miura

Authors
  1. Shunsuke Ochi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jun Miura
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jun Miura .

Editor information

Editors and Affiliations

  1. IBM Research - MIT-IBM Watson AI Lab, Massachusetts, USA

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ochi, S., Miura, J. (2023). Depth-Based In-Bed Human Pose Estimation with Synthetic Dataset Generation and Deep Keypoint Estimation. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13806. Springer, Cham. https://doi.org/10.1007/978-3-031-25075-0_45

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25075-0_45

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25074-3

  • Online ISBN: 978-3-031-25075-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation