Skip to main content
SpringerLink
Log in

3D Facial Reconstruction from a Single Image Using a Hybrid Model Based on 3DMM and Deep Learning

  • Conference paper
  • First Online:
Intelligent Human Computer Interaction (IHCI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14532))

Included in the following conference series:

  • 78 Accesses

Abstract

A fundamental challenge in computer vision is accurately modelling 3D faces from images. It facilitates the creation of immersive virtual experiences, realistic facial animations, and reliable identity verification. This research introduces an innovative approach aimed at reconstructing intricate facial attributes, encompassing shape, pose, and expression from a single input image. The proposed methodology employs a fusion of two potent techniques: 3D Morphable Models (3DMMs) and advanced Deep Learning (DL) methodologies. By integrating DL into tasks like face detection, expression analysis, and landmark extraction, the framework excels in reconstructing realistic facial attributes from single images even in diverse environments. The framework achieves compelling results in reconstructing “in-the-wild” faces, exhibiting notable fidelity while preserving essential facial characteristics. Experimental evaluations confirm the effectiveness and robustness of our approach, confirming its adaptability across various scenarios. Our research contributes to the advancement of 3D face modelling techniques, addressing the challenges of accurate reconstruction and holding promise for applications in virtual reality, facial animation, medical, security, and biometrics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Widanagamaachchi, W.N., Dharmaratne, A.T.: 3D Face Reconstruction from 2D Images. In: 2008 Digital Image Computing: Techniques and Applications, pp. 365–371. IEEE (2008). https://doi.org/10.1109/DICTA.2008.83

  2. Zollhöfer, M., et al.: State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications. Computer Graphics Forum. 37, 523–550 (2018). https://doi.org/10.1111/cgf.13382

    Article  Google Scholar 

  3. Afzal, H.M.R., Luo, S., Afzal, M.K., Chaudhary, G., Khari, M., Kumar, S.A.P.: 3D Face Reconstruction From Single 2D Image Using Distinctive Features. IEEE Access. 8, 180681–180689 (2020). https://doi.org/10.1109/ACCESS.2020.3028106

    Article  Google Scholar 

  4. Diwakar, M., Kumar, P.: 3-D Shape Reconstruction Based CT Image Enhancement. In: Handbook of Multimedia Information Security: Techniques and Applications, pp. 413–419. Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-15887-3_19

  5. Uddin, M., Manickam, S., Ullah, H., Obaidat, M., Dandoush, A.: Unveiling the Metaverse: Exploring Emerging Trends, Multifaceted Perspectives, and Future Challenges. IEEE Access. 1–1 (2023). https://doi.org/10.1109/ACCESS.2023.3281303

  6. Jha, J., et al.: Artificial intelligence and applications. In: 2023 1st International Conference on Intelligent Computing and Research Trends (ICRT), pp. 1–4. IEEE (2023). https://doi.org/10.1109/ICRT57042.2023.10146698

  7. Sharma, H., Kumar, H., Gupta, A., Shah, M.A.: Computer Vision in Manufacturing: A Bibliometric Analysis and future research propositions. Presented at the (2023)

    Google Scholar 

  8. Khari, M., Garg, A.K., Gonzalez-Crespo, R., Verdú, E.: Gesture Recognition of RGB and RGB-D Static Images Using Convolutional Neural Networks. Int. J. Interact. Multi. Artifi. Intell. 5, 22 (2019)

    Google Scholar 

  9. Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pp. 187–194 (1999)

    Google Scholar 

  10. Kittler, J., Huber, P., Feng, Z.-H., Hu, G., Christmas, W.: 3D Morphable Face Models and Their Applications. Presented at the (2016). https://doi.org/10.1007/978-3-319-41778-3_19

  11. Booth, J., Roussos, A., Ponniah, A., Dunaway, D., Zafeiriou, S.: Large Scale 3D Morphable Models. Int. J. Comput. Vis. 126, 233–254 (2018)

    Article  MathSciNet  Google Scholar 

  12. Tran, L., Liu, X.: Nonlinear 3D Face Morphable Model. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7346–7355. IEEE (2018)

    Google Scholar 

  13. Tran, L., Liu, X.: On Learning 3D Face Morphable Model from In-the-wild Images. IEEE Trans. Pattern Anal. Mach. Intell. 43, 157–171 (2019). https://doi.org/10.1109/TPAMI.2019.2927975

    Article  Google Scholar 

  14. Tran, L., Liu, F., Liu, X.: Towards High-Fidelity Nonlinear 3D Face Morphable Model. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1126–1135. IEEE (2019). https://doi.org/10.1109/CVPR.2019.00122

  15. Dai, H., Pears, N., Smith, W., Duncan, C.: Statistical modeling of craniofacial shape and texture. Int. J. Comput. Vis. 128, 547–571 (2020). https://doi.org/10.1007/s11263-019-01260-7

    Article  Google Scholar 

  16. Galanakis, S., Gecer, B., Lattas, A., Zafeiriou, S.: 3DMM-RF: convolutional radiance fields for 3D face modeling. In: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3525–3536. IEEE (2023)

    Google Scholar 

  17. Zhang, W., et al.: SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8652–8661 (2023)

    Google Scholar 

  18. Jiang, L., Zhang, J., Deng, B., Li, H., Liu, L.: 3D face reconstruction with geometry details from a single image. IEEE Trans. Image Process. 27, 4756–4770 (2018). https://doi.org/10.1109/TIP.2018.2845697

    Article  MathSciNet  Google Scholar 

  19. Chen, A., Chen, Z., Zhang, G., Mitchell, K., Yu, J.: Photo-realistic facial details synthesis from single image. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9428–9438. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00952

  20. Gecer, B., Ploumpis, S., Kotsia, I., Zafeiriou, S.: GANFIT: generative adversarial network fitting for high fidelity 3D face reconstruction. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1155–1164. IEEE (2019). https://doi.org/10.1109/CVPR.2019.00125

  21. Lattas, A., et al.: AvatarMe: Realistically Renderable 3D Facial Reconstruction “In-the-Wild.” In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 757–766. IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00084

  22. Yu, W., et al.: NOFA: NeRF-based One-shot Facial Avatar Reconstruction. In: Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings, pp. 1–12. ACM, New York, NY, USA (2023)

    Google Scholar 

  23. Bai, Z., Cui, Z., Rahim, J.A., Liu, X., Tan, P.: Deep facial non-rigid multi-view stereo. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5849–5859. IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00589

  24. Fu, K., Xie, Y., Jing, H., Zhu, J.: Fast spatial–temporal stereo matching for 3D face reconstruction under speckle pattern projection. Image Vis. Comput. 85, 36–45 (2019). https://doi.org/10.1016/j.imavis.2019.02.007

    Article  Google Scholar 

  25. Wang, X., Guo, Y., Yang, Z., Zhang, J.: Prior-Guided Multi-View 3D Head Reconstruction. IEEE Trans. Multimedia 24, 4028–4040 (2022)

    Article  Google Scholar 

  26. Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 296–301. IEEE (2009). https://doi.org/10.1109/AVSS.2009.58

  27. Guo, Y., Zhang, J., Cai, J., Jiang, B., Zheng, J.: CNN-based real-time dense face reconstruction with inverse-rendered photo-realistic face images. IEEE Trans. Pattern Anal. Mach. Intell. 41, 1294–1307 (2019). https://doi.org/10.1109/TPAMI.2018.2837742

    Article  Google Scholar 

  28. Ramamoorthi, R., Hanrahan, P.: A signal-processing framework for inverse rendering. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp. 117–128. ACM, New York, NY, USA (2001). https://doi.org/10.1145/383259.383271

  29. Ramamoorthi, R., Hanrahan, P.: An efficient representation for irradiance environment maps. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp. 497–500. ACM, New York, NY, USA (2001)

    Google Scholar 

  30. Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 815–823. IEEE (2015). https://doi.org/10.1109/CVPR.2015.7298682

  31. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3D face alignment problem? (and a Dataset of 230,000 3D Facial Landmarks). In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1021–1030. IEEE (2017)

    Google Scholar 

  32. Deng, Y., et al.: Accurate 3D face reconstruction with weakly-supervised learning: from single image to image set. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 285–295. IEEE (2019). https://doi.org/10.1109/CVPRW.2019.00038

  33. Jones, M.J., Rehg, J.M.: Statistical color models with application to skin detection. Int. J. Comput. Vis. 46, 81–96 (2002). https://doi.org/10.1023/A:1013200319198

    Article  Google Scholar 

  34. Hou, Z.-D., Kim, K.-H., Lee, D.-J., Zhang, G.-H.: Real-time markerless facial motion capture of personalized 3D real human research. Int. J. Inter. Broadcas. Comm. 14, 129–135 (2022)

    Google Scholar 

  35. OpenCV: Open Source Computer Vision Library (2015)

    Google Scholar 

  36. Johnson, J., et al.: Accelerating 3D deep learning with PyTorch3D. In: SIGGRAPH Asia 2020 Courses, p. 1. ACM, New York, NY, USA (2020). https://doi.org/10.1145/3415263.3419160

  37. Harris, C.R., et al.: Array programming with NumPy. Nature 585, 357–362 (2020)

    Article  Google Scholar 

  38. Kingma, D.P., Ba, J.: Adam: a method for Stochastic Optimization. CoRR. abs/1412.6980 (2014)

    Google Scholar 

  39. Tewari, A., et al.: MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3735–3744. IEEE (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graphic Era (Deemed to be) University, Dehradun, India

    Isha Deshmukh, Vikas Tripathi & Durgaprasad Gangodkar

Authors
  1. Isha Deshmukh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vikas Tripathi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Durgaprasad Gangodkar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Isha Deshmukh .

Editor information

Editors and Affiliations

  1. Soongsil University, Seoul, Korea (Republic of)

    Bong Jun Choi

  2. Saint Louis University, St. Louis, MO, USA

    Dhananjay Singh

  3. Indian Institute of Information Technology, Allahabad, India

    Uma Shanker Tiwary

  4. Pukyong National University, Busan, Korea (Republic of)

    Wan-Young Chung

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Deshmukh, I., Tripathi, V., Gangodkar, D. (2024). 3D Facial Reconstruction from a Single Image Using a Hybrid Model Based on 3DMM and Deep Learning. In: Choi, B.J., Singh, D., Tiwary, U.S., Chung, WY. (eds) Intelligent Human Computer Interaction. IHCI 2023. Lecture Notes in Computer Science, vol 14532. Springer, Cham. https://doi.org/10.1007/978-3-031-53830-8_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-53830-8_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-53829-2

  • Online ISBN: 978-3-031-53830-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation