Skip to main content
SpringerLink
Log in
Book cover

European Conference on Computer Vision

ECCV 2022: Computer Vision – ECCV 2022 Workshops pp 398–406Cite as

End to End Face Reconstruction via Differentiable PnP

  • Conference paper
  • First Online:

  • 1436 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13805))

Abstract

This is a challenge report of the ECCV 2022 WCPA Challenge, Face Reconstruction Track. Inside this report is a brief explanation of how we accomplish this challenge. We design a two-branch network to accomplish this task, whose roles are Face Reconstruction and Face Landmark Detection. The former outputs canonical 3D face coordinates. The latter outputs pixel coordinates, i.e. 2D mapping of 3D coordinates with head pose and perspective projection. In addition, we utilize a differentiable PnP (Perspective-n-Points) layer to finetune the outputs of the two branch. Our method achieves very competitive quantitative results on the MVP-Human dataset and wins a \(3^{rd}\) prize in the challenge.

Y. Lu—Work done during an internship in Tencent.

Y. Lu and H. Wei—Contributed equally to this work.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation.

References

  1. Bai, Z., Cui, Z., Liu, X., Tan, P.: Riggable 3D face reconstruction via in-network optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6216–6225 (June 2021)

    Google Scholar 

  2. Chen, H., Wang, P., Wang, F., Tian, W., Xiong, L., Li, H.: EPro-PnP: generalized end-to-end probabilistic perspective-n-points for monocular object pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2781–2790 (2022)

    Google Scholar 

  3. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848

  4. Feng, Y., Feng, H., Black, M.J., Bolkart, T.: Learning an animatable detailed 3D face model from in-the-wild images (2020). https://doi.org/10.48550/ARXIV.2012.04012,https://arxiv.org/abs/2012.04012

  5. Feng, Y., Wu, F., Shao, X., Wang, Y., Zhou, X.: Joint 3D face reconstruction and dense alignment with position map regression network (2018). https://doi.org/10.48550/ARXIV.1803.07835.https://arxiv.org/abs/1803.07835

  6. Guo, J., Zhu, X., Lei, Z.: 3ddfa. https://github.com/cleardusk/3DDFA (2018)

  7. Guo, J., Zhu, X., Yang, Y., Yang, F., Lei, Z., Li, S.Z.: Towards fast, accurate and stable 3D dense face alignment. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12364, pp. 152–168. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58529-7_10

    Chapter  Google Scholar 

  8. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  9. Hempel, T., Abdelrahman, A.A., Al-Hamadi, A.: 6D rotation representation for unconstrained head pose estimation. arXiv preprint arXiv:2202.12555 (2022)

  10. Kao, Y., et al.: Single-image 3D face reconstruction under perspective projection. arXiv preprint arXiv:2205.04126 (2022)

  11. Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: Guyon, I.,et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017)

    Google Scholar 

  12. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2015)

    Google Scholar 

  13. Liu, R., et al.: An intriguing failing of convolutional neural networks and the coordconv solution. ArXiv abs/1807.03247 (2018)

    Google Scholar 

  14. Wood, E., et al.: 3D face reconstruction with dense landmarks (2022). https://doi.org/10.48550/ARXIV.2204.02776,https://arxiv.org/abs/2204.02776

Download references

Author information

Authors and Affiliations

  1. State University of New York at Buffalo, Buffalo, USA

    Yiren Lu

  2. Tencent, Shenzhen, China

    Huawei Wei

Authors
  1. Yiren Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huawei Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yiren Lu .

Editor information

Editors and Affiliations

  1. IBM Research AI and MIT-IBM Watson AI Lab, Haifa, Israel

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lu, Y., Wei, H. (2023). End to End Face Reconstruction via Differentiable PnP. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13805. Springer, Cham. https://doi.org/10.1007/978-3-031-25072-9_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25072-9_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25071-2

  • Online ISBN: 978-3-031-25072-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation