Abstract
This paper presents a monocular 3D human pose estimation approach for virtual character skeleton retargeting with monocular visual equipment. First, the 2D human pose is achieved by using the OpenPose method from the continuous video frames collected by the monocular camera, and the corresponding 3D human pose is estimated by fusing and constructing the depth-channel pose estimation network. The pose filter network is next designed to smooth and optimize the 3D human pose estimation through a sliding window strategy. The human pose skeleton retargeting and optimizer methods are then proposed to support video motion capture applications and virtual character skeleton retargeting for animation based on the bone direction vectors and the re-projection error of joint points. Finally, the performance of the proposed approach is verified on Human3.6 M dataset, and the results show that the mean per-joint position error in the public dataset is 40.5 mm, which is lower than that of the multiple benchmark methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The datasets generated during and analysed during the current study are available from the corresponding author on reasonable request.
References
Cao Z, Hidalgo G, Simon T et al (2021) OpenPose: realtime multi-person 2D pose estimation using part affinity fields. IEEE Trans Pattern Anal Mach Intell 43(1):172–186. https://doi.org/10.1109/TPAMI.2019.2929257
Chen Y, Shen C, Chen H et al (2020) Adversarial learning of structure-aware fully convolutional networks for landmark localization. IEEE Trans Pattern Anal Mach Intell 42(7):1654–1669. https://doi.org/10.1109/TPAMI.2019.2901875
Chen YC, Piccirilli M, Piramuthu R et al (2021) Self-attentive 3D human pose and shape estimation from videos. Comput vis Image Underst 213(2021):103305–103318. https://doi.org/10.1016/j.cviu.2021.103305
Dabral R, Mundhada A, Kusupati U et al (2018) Learning 3D human pose from structure and motion. Eur Conf Comput Vision 11213:679–696. https://doi.org/10.1007/978-3-030-01240-3_41
Gu R, Wang G, Jiang Z et al (2020) Multi-person hierarchical 3D pose estimation in natural videos. IEEE Trans Circuits Syst Video Technol 30(11):4245–4257. https://doi.org/10.1109/TCSVT.2019.2953678
Ionescu C, Papava D, Olaru V et al (2014) Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Trans Pattern Anal Mach Intell 36(7):1325–1339. https://doi.org/10.1109/TPAMI.2013.248
Jain DK, Zareapoor M, Jain R et al (2020) GAN-Poser: an improvised bidirectional GAN model for human motion prediction. Neural Comput Appl 32(18):14579–14591. https://doi.org/10.1007/s00521-020-04941-4
Katircioglu I, Tekin B, Salzmann M et al (2018) Learning latent representations of 3D human pose with deep neural networks. Int J Comput Vision 126(12):1326–1341. https://doi.org/10.1007/s11263-018-1066-6
Kim DY, Chang JY (2021) Attention-based 3D human pose sequence refinement network. Sensors 21(13):4752–4770. https://doi.org/10.3390/s21134572
Li S, Zhang W, Chan A (2017) Maximum-margin structured learning with deep networks for 3D human pose estimation. Int J Comput Vision 122:149–168. https://doi.org/10.1007/s11263-016-0962-x
Li Y, Li K, Jiang S et al (2020) Geometry-driven self-supervised method for 3D human pose estimation. AAAI Conf Artif Intell. https://doi.org/10.1609/aaai.v34i07.6808
Loper M, Mahmood N, Romero J et al (2015) SMPL: a skinned multi-person linear model. ACM Trans Graph 34(6):1–16. https://doi.org/10.1145/2816795.2818013
Martinez J, Hossain R, Romero J et al (2017) A simple yet effective baseline for 3D human pose estimation. IEEE Int Conf Comput Vision. https://doi.org/10.1109/ICCV.2017.288
Mehta D, Sridhar S, Sotnychenko O et al (2017) VNect: real-time 3D human pose estimation with a single RGB camera. ACM Transa Graph 36(4):1–14. https://doi.org/10.1145/3072959.3073596
Moon G, Chang JY, Lee KM (2019) Camera distance-aware top-down approach for 3D multi-person pose estimation from a single RGB image. IEEE Int Conf Comput Vision. https://doi.org/10.1109/ICCV.2019
Naveen P, Sivakumar P (2021) Adaptive morphological and bilateral filtering with ensemble convolutional neural network for pose-invariant face recognition. J Ambient Intell Humaniz Comput 12:10023–10033. https://doi.org/10.1007/s12652-020-02753-x
Rogez G, Weinzaepfel P, Schmid C (2020) LCR-Net++: multi-person 2D and 3D pose detection in natural images. IEEE Trans Pattern Anal Mach Intell 42(5):1146–1161. https://doi.org/10.1109/TPAMI.2019.2892985
Selva J, Johansen AS, Escalera S et al (2023) Video transformers: a survey. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2023.3243465
Shi M, Aberman K, Aristidou A et al (2021) MotioNet: 3D human motion reconstruction from monocular video with skeleton consistency. ACM Trans Graph 40(1):1–15. https://doi.org/10.1145/3407659
Verma P, Srivastava R (2020) Three stage deep network for 3D human pose reconstruction by exploiting spatial and temporal data via its 2D pose. J vis Commun Image Represent 71:102866–102876. https://doi.org/10.1016/j.jvcir.2020.102866
Wang M, Qiu F, Liu W et al (2020) Monocular human pose and shape reconstruction using part differentiable rendering. Comput Graph Forum 39(7):351–362. https://doi.org/10.1111/cgf.14150
Xia H, Xiao M (2020) 3D human pose estimation with generative adversarial networks. IEEE Access 8:206198–206206. https://doi.org/10.1109/ACCESS.2020.3037829
Yang A, Ren H, Fei M, Naeem W (2021b) Multi-person vision tracking approach based on human body localization features. Adv Manuf 9(4):496–508. https://doi.org/10.1007/s40436-021-00363-0
Yang D, Kim D, Lee SH (2021c) LoBSTr: Real-time lower-body pose prediction from sparse upper-body tracking signals. Comput Graph Forum 40(2):265–275. https://doi.org/10.1111/cgf.142631
Yang A, Chen Y, Naeem W, Fei M, Chen L (2021a) Humanoid motion planning of robotic arm based on human arm action feature and reinforcement learning. Mechatronics 78:102630. https://doi.org/10.1016/j.mechatronics.2021.102630
Zheng X, Chen X, Lu X (2020) A joint relationship aware neural network for single-image 3D human pose Estimation. IEEE Trans Image Process 29:4747–4758. https://doi.org/10.1109/TIP.2020.2972104
Zhou X, Zhu M, Leonardos S et al (2017) Sparse representation for 3D shape estimation: a convex relaxation approach. IEEE Trans Pattern Anal Mach Intell 39(8):1648–1661. https://doi.org/10.1109/TPAMI.2016.2605097
Acknowledgements
This work is supported by Natural Science Foundation of Shanghai under Grant 22ZR1424200.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that we have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary file1 (MP4 1755 KB)
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Yang, A., Liu, G., Naeem, W. et al. A monocular 3D human pose estimation approach for virtual character skeleton retargeting. J Ambient Intell Human Comput 14, 9563–9574 (2023). https://doi.org/10.1007/s12652-023-04629-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-023-04629-2