Skip to main content

Skeleton-Free Pose Transfer for Stylized 3D Characters

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13662))

Included in the following conference series:

  • 2682 Accesses

Abstract

We present the first method that automatically transfers poses between stylized 3D characters without skeletal rigging. In contrast to previous attempts to learn pose transformations on fixed or topology-equivalent skeleton templates, our method focuses on a novel scenario to handle skeleton-free characters with diverse shapes, topologies, and mesh connectivities. The key idea of our method is to represent the characters in a unified articulation model so that the pose can be transferred through the correspondent parts. To achieve this, we propose a novel pose transfer network that predicts the character skinning weights and deformation transformations jointly to articulate the target character to match the desired pose. Our method is trained in a semi-supervised manner absorbing all existing character data with paired/unpaired poses and stylized shapes. It generalizes well to unseen stylized characters and inanimate objects. We conduct extensive experiments and demonstrate the effectiveness of our method on this novel task.

Z. Liao—Work done during an internship at Adobe.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Mixamo (2022). http://www.mixamo.com/

  2. Aberman, K., Li, P., Lischinski, D., Sorkine-Hornung, O., Cohen-Or, D., Chen, B.: Skeleton-aware networks for deep motion retargeting. ACM Trans. Graph. (TOG) 39(4), 1–14 (2020)

    Article  Google Scholar 

  3. Al Borno, M., Righetti, L., Black, M.J., Delp, S.L., Fiume, E., Romero, J.: Robust physics-based motion retargeting with realistic body shapes. In: Computer Graphics Forum, vol. 37, pp. 81–92. Wiley Online Library (2018)

    Google Scholar 

  4. Aristidou, A., Lasenby, J.: FABRIK: a fast, iterative solver for the inverse kinematics problem. Graph. Models 73(5), 243–260 (2011)

    Article  Google Scholar 

  5. Avril, Q., et al.: Animation setup transfer for 3D characters. In: Computer Graphics Forum, vol. 35, pp. 115–126. Wiley Online Library (2016)

    Google Scholar 

  6. Baran, I., Popović, J.: Automatic rigging and animation of 3D characters. ACM Trans. Graph. (TOG) 26(3), 72–es (2007)

    Google Scholar 

  7. Baran, I., Vlasic, D., Grinspun, E., Popović, J.: Semantic deformation transfer. In: ACM SIGGRAPH 2009 Papers, pp. 1–6 (2009)

    Google Scholar 

  8. Ben-Chen, M., Weber, O., Gotsman, C.: Spatial deformation transfer. In: Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 67–74 (2009)

    Google Scholar 

  9. Besl, P.J., McKay, N.D.: Method for registration of 3D shapes. In: Sensor Fusion IV: Control Paradigms and Data Structures (1992)

    Google Scholar 

  10. Bhatnagar, B.L., Tiwari, G., Theobalt, C., Pons-Moll, G.: Multi-garment net: learning to dress 3D people from images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5420–5430 (2019)

    Google Scholar 

  11. Bogo, F., Romero, J., Loper, M., Black, M.J.: FAUST: dataset and evaluation for 3D mesh registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3794–3801 (2014)

    Google Scholar 

  12. Choi, K.J., Ko, H.S.: Online motion retargetting. J. Vis. Comput. Animat. 11(5), 223–235 (2000)

    Article  MATH  Google Scholar 

  13. Fernandez-Labrador, C., Chhatkuli, A., Paudel, D.P., Guerrero, J.J., Demonceaux, C., Gool, L.V.: Unsupervised learning of category-specific symmetric 3D keypoints from point sets. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 546–563. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58595-2_33

    Chapter  Google Scholar 

  14. Gao, L., Lai, Y.K., Yang, J., Zhang, L.X., Xia, S., Kobbelt, L.: Sparse data driven mesh deformation. IEEE TVCG 27, 2085–2100 (2019)

    Google Scholar 

  15. Gao, L., et al.: Automatic unpaired shape deformation transfer. ACM Trans. Graph. (TOG) 37(6), 1–15 (2018)

    Google Scholar 

  16. Gleicher, M.: Retargetting motion to new characters. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, pp. 33–42 (1998)

    Google Scholar 

  17. Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: 3D-CODED: 3D correspondences by deep deformation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 235–251. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_15

    Chapter  Google Scholar 

  18. Hung, W.C., Jampani, V., Liu, S., Molchanov, P., Yang, M.H., Kautz, J.: SCOPS: self-supervised co-part segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 869–878 (2019)

    Google Scholar 

  19. Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6M: large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1325–1339 (2013)

    Article  Google Scholar 

  20. Jacobson, A., Deng, Z., Kavan, L., Lewis, J.P.: Skinning: real-time shape deformation (full text not available). In: ACM SIGGRAPH 2014 Courses, p. 1 (2014)

    Google Scholar 

  21. Jakab, T., Gupta, A., Bilen, H., Vedaldi, A.: Unsupervised learning of object landmarks through conditional image generation. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

    Google Scholar 

  22. Jakab, T., Tucker, R., Makadia, A., Wu, J., Snavely, N., Kanazawa, A.: KeypointDeformer: unsupervised 3D keypoint discovery for shape control. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12783–12792 (2021)

    Google Scholar 

  23. Kavan, L.: Direct skinning methods and deformation primitives. In: ACM SIGGRAPH Courses (2014)

    Google Scholar 

  24. Lee, J., Shin, S.Y.: A hierarchical approach to interactive motion editing for human-like figures. In: Proceedings of the 26th Annual Conference on Computer graphics and Interactive Techniques, pp. 39–48 (1999)

    Google Scholar 

  25. Li, P., Aberman, K., Hanocka, R., Liu, L., Sorkine-Hornung, O., Chen, B.: Learning skeletal articulations with neural blend shapes. ACM Trans. Graph. (TOG) 40(4), 1–15 (2021)

    Article  Google Scholar 

  26. Lim, J., Chang, H.J., Choi, J.Y.: PMnet: learning of disentangled pose and movement for unsupervised motion retargeting. In: BMVC, vol. 2, p. 7 (2019)

    Google Scholar 

  27. Liu, L., Zheng, Y., Tang, D., Yuan, Y., Fan, C., Zhou, K.: NeuroSkinning: automatic skin binding for production characters with deep graph networks. ACM TOG 38, 1–12 (2019)

    Google Scholar 

  28. Liu, M., Sung, M., Mech, R., Su, H.: DeepMetaHandles: learning deformation meta-handles of 3D meshes with biharmonic coordinates. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12–21 (2021)

    Google Scholar 

  29. Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Trans. Graph. (TOG) 34(6), 1–16 (2015)

    Article  Google Scholar 

  30. Mahmood, N., Ghorbani, N., Troje, N.F., Pons-Moll, G., Black, M.J.: AMASS: archive of motion capture as surface shapes. In: International Conference on Computer Vision, pp. 5442–5451 (2019)

    Google Scholar 

  31. Musoni, P., Marin, R., Melzi, S., Castellani, U.: Reposing and retargeting unrigged characters with intrinsic-extrinsic transfer. In: Smart Tools and Applications in Graphics (2021)

    Google Scholar 

  32. Poirier, M., Paquette, E.: Rig retargeting for 3D animation. In: Graphics Interface, pp. 103–110 (2009)

    Google Scholar 

  33. Reed, S.E., Zhang, Y., Zhang, Y., Lee, H.: Deep visual analogy-making. In: Proceedings of the NeurIPS (2015)

    Google Scholar 

  34. Rhodin, H., et al.: Generalizing wave gestures from sparse examples for real-time character control. ACM Trans. Graph. 34(6), 1–12 (2015)

    Article  Google Scholar 

  35. Saito, S., Yang, J., Ma, Q., Black, M.J.: SCANimate: weakly supervised learning of skinned clothed avatar networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2886–2897 (2021)

    Google Scholar 

  36. Shi, R., Xue, Z., You, Y., Lu, C.: Skeleton merger: an unsupervised aligned keypoint detector. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 43–52 (2021)

    Google Scholar 

  37. Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., Sebe, N.: Animating arbitrary objects via deep motion transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2377–2386 (2019)

    Google Scholar 

  38. Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., Sebe, N.: First order motion model for image animation. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  39. Siarohin, A., Roy, S., Lathuilière, S., Tulyakov, S., Ricci, E., Sebe, N.: Motion-supervised co-part segmentation. arXiv preprint arXiv:2004.03234 (2020)

  40. Siarohin, A., Woodford, O.J., Ren, J., Chai, M., Tulyakov, S.: Motion representations for articulated animation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13653–13662 (2021)

    Google Scholar 

  41. Song, C., Wei, J., Li, R., Liu, F., Lin, G.: 3D pose transfer with correspondence learning and mesh refinement. In: Advances in Neural Information Processing Systems, vol. 34 (2021)

    Google Scholar 

  42. Sorkine, O., Alexa, M.: As-rigid-as-possible surface modeling. In: Symposium on Geometry Processing (2007)

    Google Scholar 

  43. Sumner, R.W., Popović, J.: Deformation transfer for triangle meshes. ACM Trans. Graph. (TOG) 23(3), 399–405 (2004)

    Article  Google Scholar 

  44. Tak, S., Ko, H.S.: A physically-based motion retargeting filter. ACM Trans. Graph. (TOG) 24(1), 98–117 (2005)

    Article  Google Scholar 

  45. Tan, Q., Gao, L., Lai, Y.K., Xia, S.: Variational autoencoders for deforming 3D mesh models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5841–5850 (2018)

    Google Scholar 

  46. Villegas, R., Ceylan, D., Hertzmann, A., Yang, J., Saito, J.: Contact-aware retargeting of skinned motion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9720–9729 (2021)

    Google Scholar 

  47. Villegas, R., Yang, J., Ceylan, D., Lee, H.: Neural kinematic networks for unsupervised motion retargetting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8639–8648 (2018)

    Google Scholar 

  48. Wang, J., et al.: Neural pose transfer by spatially adaptive instance normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5831–5839 (2020)

    Google Scholar 

  49. Xu, Z., Zhou, Y., Kalogerakis, E., Landreth, C., Singh, K.: RigNet: neural rigging for articulated characters. arXiv preprint arXiv:2005.00559 (2020)

  50. Xu, Z., Zhou, Y., Kalogerakis, E., Singh, K.: Predicting animation skeletons for 3D articulated models via volumetric nets. In: 3DV (2019)

    Google Scholar 

  51. Yamane, K., Ariki, Y., Hodgins, J.: Animating non-humanoid characters with human motion data. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 169–178 (2010)

    Google Scholar 

  52. Yang, J., Gao, L., Lai, Y.K., Rosin, P.L., Xia, S.: Biharmonic deformation transfer with automatic key point selection. Graph. Models 98, 1–13 (2018)

    Article  MathSciNet  Google Scholar 

  53. Yang, J., Gao, L., Tan, Q., Huang, Y., Xia, S., Lai, Y.K.: Multiscale mesh deformation component analysis with attention-based autoencoders. arXiv preprint arXiv:2012.02459 (2020)

  54. Zhang, X., Bhatnagar, B.L., Starke, S., Guzov, V., Pons-Moll, G.: COUCH: towards controllable human-chair interactions. In: European Conference on Computer Vision (ECCV). Springer (2022)

    Google Scholar 

  55. Zhou, K., Bhatnagar, B.L., Pons-Moll, G.: Unsupervised shape and pose disentanglement for 3D meshes. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12367, pp. 341–357. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58542-6_21

    Chapter  Google Scholar 

  56. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

    Google Scholar 

Download references

Acknowledgement

This work is funded by a gift from Adobe Research and the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 409792180 (Emmy Noether Programme, project: Real Virtual Humans). Gerard Pons-Moll is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zhouyingcheng Liao .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 210 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liao, Z., Yang, J., Saito, J., Pons-Moll, G., Zhou, Y. (2022). Skeleton-Free Pose Transfer for Stylized 3D Characters. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13662. Springer, Cham. https://doi.org/10.1007/978-3-031-20086-1_37

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20086-1_37

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20085-4

  • Online ISBN: 978-3-031-20086-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics