Skip to main content
Springer Nature Link
Log in

Human-Robot Pose Tracking Based on CNN with Color and Geometry Aggregation

  • Conference paper
  • First Online:
Social Robotics (ICSR + InnoBiz 2024)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 15170))

Included in the following conference series:

  • 13 Accesses

Abstract

Accurately tracking the robotic arm and human joints is crucial to ensure safety during human-robot interaction. However, traditional pose tracking methods often exhibit insufficient performance and robustness in complex environments. The variations in the robotic arm’s environment and posture make it challenging for traditional methods to accurately capture the positions and posture of its joints. Specifically, when addressing challenges such as high similarity, occlusion, background complexity, and joint recognition failures, they often struggle to provide reliable and accurate results. To address these challenges, this paper proposes a human-robot pose tracking algorithm based on a new convolutional neural network model. To enhance detection accuracy, an improved color detection module is introduced to resolve joint misclassification. A geometric perception module is designed to accurately locate joints even under occlusion. Additionally, innovative iEMA and DBB modules are incorporated. The iEMA module employs edge detection technology to dynamically adjust thresholds for correct matching by improving the edge matching process. The DBB module refines the boundary box parameters for precise localization by introducing adaptive bounding box updates in real-time. This algorithm also integrates human pose recognition, enabling real-time pose recognition, thereby facilitating more intelligent and natural human-robot interaction. The algorithm has been rigorously evaluated on a custom-designed robotic arm platform. Experimental results validate the algorithm’s effectiveness and feasibility.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Kang, S., Kim, M., Kim, K.: Safety monitoring for human robot collaborative workspaces. In: 2019 19th International Conference on Control, Automation and Systems (ICCAS), pp. 1192–1194. IEEE (2019)

    Google Scholar 

  2. Iqbal, K.F., Kanazawa, A., Ottaviani, S.R., Kinugawa, J., Kosuge, K.: A real-time motion planning scheme for collaborative robots using HRI-based cost function. Int. J. Mechatron. Autom. 8(1), 42–52 (2021)

    Article  Google Scholar 

  3. Srinivasan, K., Porkumaran, K., Sainarayanan, G.: A new approach for human activity analysis through identification of body parts using skin colour segmentation. Int. J. Signal Imaging Syst. Eng. 3(2), 93–104 (2010)

    Google Scholar 

  4. Saravanakumar, S., Vadivel, A., Ahmed, C.S.: Object tracking and failure recovery in motion video sequences using the properties of HSV colour space. Int. J. Signal Imaging Syst. Eng. 5(1), 29–42 (2012)

    MATH  Google Scholar 

  5. Kshirsagar, K.P., Doye, D.D.: Comparing key frame selection for one-two hand gesture recognition using different methods. Int. J. Signal Imaging Syst. Eng. 8(5), 273–285 (2015)

    Google Scholar 

  6. Amri, S., Barhoumi, W., Zagrouba, E.: Detection and matching of multiple occluded moving people for human tracking in colour video sequences. Int. J. Signal Imaging Syst. Eng. 4(3), 153–163 (2011)

    MATH  Google Scholar 

  7. Hu, Z., Zhang, C., Wang, X., Ge, A.: Light-adaptive human body key point detection algorithm based on multi-source information fusion. Sensors 24(10), 3021 (2024)

    Article  MATH  Google Scholar 

  8. Li, X.: Study on volleyball-movement pose recognition based on joint point sequence. Comput. Intell. Neurosci. 2023(1), 2198495 (2023)

    Article  Google Scholar 

  9. Szabó, R., Gontean, A.: Robotic arm detection in the 2D space. In: 2014 IEEE 20th International Symposium for Design and Technology in Electronic Packaging (SIITME), pp. 215–220. IEEE (2014)

    Google Scholar 

  10. Mišeikis, J., Brijačak, I., Yahyanejad, S., Glette, K., Elle, O.J., Torresen, J.: Two-stage transfer learning for heterogeneous robot detection and 3D joint position estimation in a 2D camera image using CNN. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 8883–8889. IEEE (2019)

    Google Scholar 

  11. Ouyang, D., et al.: Efficient multi-scale attention module with cross-spatial learning. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE (2023)

    Google Scholar 

  12. Zhang, J., et al.: Rethinking mobile block for efficient attention-based models. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1389–1400. IEEE Computer Society (2023)

    Google Scholar 

  13. Ding, X., Zhang, X., Han, J., Ding, G.: Diverse branch block: building a convolution as an inception-like unit. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10886–10895 (2021)

    Google Scholar 

  14. Mayyas, K., Aboulnasr, T.: Leaky LMS algorithm: MSE analysis for gaussian data. IEEE Trans. Signal Process. 45(4), 927–934 (1997)

    Article  MATH  Google Scholar 

  15. Mao, A., Mohri, M., Zhong, Y.: Cross-entropy loss functions: theoretical analysis and applications. In: International Conference on Machine Learning, pp. 23803–23828. PMLR (2023)

    Google Scholar 

  16. Kendall, A., Cipolla, R.: Geometric loss functions for camera pose regression with deep learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5974–5983 (2017)

    Google Scholar 

  17. Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016, Proceedings, Part VIII 14, pp. 483–499. Springer (2016)

    Google Scholar 

  18. Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703 (2019)

    Google Scholar 

  19. Zhao, Z., Song, A., Zheng, S., Xiong, Q., Guo, J.: DSC-HRNet: a lightweight teaching pose estimation model with depthwise separable convolution and deep high-resolution representation learning in computer-aided education. Int. J. Inf. Technol. 15(5), 2373–2385 (2023)

    Google Scholar 

  20. Cao, Z., Simon, T., Wei, S.-E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291–7299 (2017)

    Google Scholar 

  21. Wu, Y., Jiang, L., Yang, Y.: Revisiting embodiedqa: a simple baseline and beyond. IEEE Trans. Image Process. 29, 3984–3992 (2020)

    Article  MATH  Google Scholar 

Download references

Acknowledgments

This work was supported by Guangdong Basic and Applied Basic Research Foundation (2023A1515011363), the National Natural Science Foundation of China (62273332), the Youth Innovation Promotion Association of Chinese Academy of Sciences (2022201), and Liaoning Applied Basic Research Foundation (2023JH26/10300028).

Author information

Authors and Affiliations

  1. Shenyang University of Technology, Shenyang, 110020, China

    Yue Xu & Yinlong Zhang

  2. Guangzhou Institute of Industrial Intelligence, Guangzhou, 511458, China

    Yinlong Zhang

  3. State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, 110016, China

    Yinlong Zhang, Shuai Liu, Yuanhao Liu & Wei Liang

  4. Key Laboratory of Networked Control Systems, Chinese Academy of Sciences, Shenyang, 110016, China

    Yinlong Zhang, Shuai Liu, Yuanhao Liu & Wei Liang

  5. University of Chinese Academy of Sciences, Beijing, 100049, China

    Yinlong Zhang, Shuai Liu, Yuanhao Liu & Wei Liang

  6. Department of Computer Science, The University of Alabama, Tuscaloosa, AL, 35487, USA

    Hongsheng He

Authors
  1. Yue Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yinlong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuai Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuanhao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hongsheng He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yinlong Zhang or Shuai Liu .

Editor information

Editors and Affiliations

  1. The Chinese University of Hong Kong, Shenzhen, China

    Haizhou Li

  2. University of Bremen, Bremen, Germany

    Tanja Schultz

  3. Shenzhen Institute of Advanced Technology, Shenzhen, China

    Yalei Bi

  4. The Chinese University of Hong Kong, Shenzhen, China

    Jian Zhu

  5. The University of Alabama, Tuscaloosa, AL, USA

    Hongsheng He

  6. The Hong Kong University of Science, Guangzhou, China

    Jun Ma

  7. National University of Singapore, Singapore, Singapore

    Siqi Cai

  8. Qingdao University, Qingdao, China

    Wanyue Jiang

  9. National University of Singapore, Singapore, Singapore

    Shuzhi Sam Ge

Ethics declarations

Disclosure of Interests

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, Y., Zhang, Y., Liu, S., Liu, Y., Liang, W., He, H. (2025). Human-Robot Pose Tracking Based on CNN with Color and Geometry Aggregation. In: Li, H., et al. Social Robotics. ICSR + InnoBiz 2024. Lecture Notes in Computer Science(), vol 15170. Springer, Singapore. https://doi.org/10.1007/978-981-96-1151-5_13

Download citation

  • DOI: https://doi.org/10.1007/978-981-96-1151-5_13

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-96-1150-8

  • Online ISBN: 978-981-96-1151-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics