Skip to main content
Springer Nature Link
Log in

Robust and Real-Time Face Swapping Based on Face Segmentation and CANDIDE-3

  • Conference paper
  • First Online:
PRICAI 2018: Trends in Artificial Intelligence (PRICAI 2018)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11013))

Included in the following conference series:

Abstract

Despite some successes have been made in face swapping research, face swapping is still not robust and real-time enough. In this paper, a robust and real-time method for face swapping based on face segmentation and CANDIDE-3 is proposed. We implement our method through four steps: face detection, face alignment, modelling and swapping. We test our method on three publicly available datasets and some videos, and results show that our method can effectively improve the robust and real-time performance of face swapping.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Blanz, V., Scherbaum, K., Vetter, T., Seidel, H.P.: Exchanging faces in images. In: Computer Graphics Forum, pp. 669–676 (2004)

    Article  Google Scholar 

  2. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., Nayar, S.K.: Face swapping: automatically replacing faces in photographs. In: ACM SIGGRAPH, p. 39 (2008)

    Google Scholar 

  3. Yang, F., Wang, J., Shechtman, E., Bourdev, L., Metaxas, D.: Expression flow for 3D-aware face component transfer. Acm Trans. Graph. 30(4), 1–10 (2011)

    Article  Google Scholar 

  4. Lin, Y., Wang, S., Lin, Q., Tang, F.: Face swapping under large pose variations: a 3D model based approach. In: 2012 IEEE International Conference on Multimedia and Expo (ICME), pp. 333–338. IEEE (2012)

    Google Scholar 

  5. Nirkin, Y., Masi, I., Tran, A.T., Hassner, T., Medioni, G.: On face segmentation, face swapping, and face perception. arXiv preprint arXiv:1704.06729 (2017)

  6. De La Hunty, M., Asthana, A., Goecke, R.: Linear facial expression transfer with active appearance models. In: 2010 20th International Conference on Pattern Recognition (ICPR), pp. 3789–3792. IEEE (2010)

    Google Scholar 

  7. Zhu, J., Van Gool, L., Hoi, S.C.: Unsupervised face alignment by robust nonrigid mapping. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 1265–1272. IEEE (2009)

    Google Scholar 

  8. Liu, S., Yang, J., Huang, C., Yang, M.H.: Multi-objective convolutional learning for face labeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3451–3459 (2015)

    Google Scholar 

  9. Qian, K., Wang, B., Chen, H.: Automatic flexible face replacement with no auxiliary data. Comput. Graph. 45, 64–74 (2014)

    Article  Google Scholar 

  10. King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)

    Google Scholar 

  11. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874 (2014)

    Google Scholar 

  12. Nguyen, A., Simard-Meilleur, A., Berthiaume, C., Godbout, R., Mottron, L.: Head circumference in canadian male adults: development of a normalized chart. Int. J. Morphol. 30(4), 1474–1480 (2012)

    Article  Google Scholar 

  13. Ahlberg, J.: CANDIDE-3 - an updated parameterised face. Rinsho Byori Jpn. J. Clin. Pathol. 48(3), 385–388 (2001)

    Google Scholar 

  14. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

    Google Scholar 

  15. Burgos-Artizzu, X.P., Perona, P., Dollar, P.: Robust face landmark estimation under occlusion. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1513–1520 (2013)

    Google Scholar 

  16. Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: IEEE International Conference on Computer Vision Workshops, pp. 397–403 (2014)

    Google Scholar 

  17. Learned-Miller, E., Huang, G.B., Roychowdhury, A., Li, H., Hua, G.: Labeled Faces in the Wild: A Survey. Springer International Publishing, Heidelberg (2016)

    Google Scholar 

  18. Reinhard, E., Adhikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graph. Appl. 21(5), 34–41 (2001)

    Article  Google Scholar 

  19. Jia, Y., et al.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675–678. ACM (2014)

    Google Scholar 

  20. Bradski, G.: The opencv library. Dr. Dobbs J. Softw. Tools Prof. Program. 25(11), 120–123 (2000)

    Google Scholar 

  21. Jia, X., Yang, H., Chan, K.P., Patras, I.: Structured semi-supervised forest for facial landmarks localization with face mask reasoning. In: BMVC (2014)

    Google Scholar 

  22. Ghiasi, G., Fowlkes, C.: Using segmentation to predict the absence of occluded parts. In: British Machine Vision Conference, pp. 22.1–22.12 (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Network Technology, Beijing University of Posts and Telecommunications, Beijing, China

    Haosen Wang, Dongliang Xie & Lu Wei

Authors
  1. Haosen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dongliang Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lu Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haosen Wang .

Editor information

Editors and Affiliations

  1. Southeast University, Nanjing, China

    Xin Geng

  2. University of Tasmania, Hobart, Tasmania, Australia

    Byeong-Ho Kang

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, H., Xie, D., Wei, L. (2018). Robust and Real-Time Face Swapping Based on Face Segmentation and CANDIDE-3. In: Geng, X., Kang, BH. (eds) PRICAI 2018: Trends in Artificial Intelligence. PRICAI 2018. Lecture Notes in Computer Science(), vol 11013. Springer, Cham. https://doi.org/10.1007/978-3-319-97310-4_38

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-97310-4_38

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-97309-8

  • Online ISBN: 978-3-319-97310-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics