Skip to main content
SpringerLink
Log in

Replacement of Facial Parts in Images

  • Conference paper
  • First Online:
Next Generation Computer Animation Techniques (AniNex 2017)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10582))

  • 1289 Accesses

Abstract

It is interesting to edit facial appearance in images to create a desirable facial shape of persons. In this paper, we propose a novel method to modify facial appearance by replacing facial parts between arbitrarily paired images. To this end, our method consists of face segmentation, face reconstruction, mesh deformation and image editing. Given one source and one target image, the target image is first segmented into the front facial region and background image. Secondly, 3D facial models and relevant scene parameters are estimated from both images. Thirdly, the target facial part is replaced with the selected source part on the 3D mesh. Then, the new replaced 3D face is rendered into a facial image. Finally, the new facial image is generated by seamlessly blending the rendered image and background image. The main advantage of this method is that we transfer facial geometric information between images using 3D model, which can deal with arbitrarily paired images with the different facial viewpoint. We present several experimental results to show the effectiveness of our method and comparison with those existing methods to demonstrate that our method is more advantageous and flexible in terms of practical applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Chou, J.K., Yang, C.K., Gong, S.D.: Face-off: automatic alteration of facial features. Multimedia Tools Appl. 56(3), 569–596 (2012)

    Article  Google Scholar 

  2. Klum, S., Han, H., Jain, A.K., Klara, B.: Sketch based face recognition: Forensic vs. Composite sketches. In: 2013 International Conference on Biometrics (ICB), pp. 1–8. IEEE, Madrid, Spain (2013)

    Google Scholar 

  3. Google Street View. http://maps.google.com/help/maps/streetview

  4. Blanz, V., Scherbaum, K., Vetter, T., Seidel, H.P.: Exchanging faces in images. Comput. Graph. Forum 23(3), 669–676 (2004)

    Article  Google Scholar 

  5. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., Nayar, S.K.: Face Swapping: automatically replacing faces in photographs. ACM Trans. Graph. (TOG) 27(3), 39:1–39:8 (2008)

    Article  Google Scholar 

  6. Kemelmacher-Shlizerman, I.: Transfiguring portraits. ACM Trans. Graph. (TOG) 35(4), 94:1–94:8 (2016)

    Article  Google Scholar 

  7. Afifi, M., Hussain, K.F., Ibrahim, H.M., Omar, N.M., Video face replacement system using a modified Poisson blending technique. In: 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), pp. 205–210. IEEE, Kuching, Malaysia (2014)

    Google Scholar 

  8. Nirkin, Y., Masi, I., Tran, A. T, Hassner, T., Medioni, G.: On Face Segmentation, Face Swapping, and Face Perception. arXiv preprint arXiv:1704.06729, (2017)

  9. Liao, Q., Jin, X., Zeng, W.: Enhancing the symmetry and proportion of 3D face geometry. IEEE Trans. Visual Comput. Graph. 18(10), 1704–1716 (2012)

    Article  Google Scholar 

  10. Zhao, H., Jin, X., Huang, X., Chai, M., Zhou, K.: Parametric weight-change reshaping for portrait images. IEEE Comput. Graph. Appl. 36 (2016)

    Google Scholar 

  11. Best-Rowden, L., Han, H., Otto, C., Klare, B.F., Jain, A.K.: Unconstrained face recognition: identifying a person of interest from a media collection. IEEE Trans. Inf. Forensics Secur. 9(12), 2144–2157 (2014)

    Article  Google Scholar 

  12. Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., Niebner, M.: Face2Face: real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395. IEEE, Las Vegas, NV, USA (2016)

    Google Scholar 

  13. Cao, C., Weng, Y., Lin, S., Zhou, K.: 3D shape regression for real-time facial animation. ACM Trans. Graph. (TOG) 32(4), 41:1–41:10 (2013)

    Article  MATH  Google Scholar 

  14. Li, H., Yu, J., Ye, Y., Bregler, C.: Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. (TOG) 32(4), 42:1–42:10 (2013)

    MATH  Google Scholar 

  15. Paysan, P., Knothe, R., Amberg, B.: A 3D face model for pose and illumination invariant face recognition. In: 6th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS 2009), pp. 296–301. IEEE, Genova, Italy (2009)

    Google Scholar 

  16. Jacobson, A., Tosun, E., Sorkine, O.: Mixed finite elements for variational surface modeling. Comput. Graph. Forum 29(5), 1565–1574 (2010)

    Article  Google Scholar 

  17. Wang, H., Cao, J., Liu, X., Wang, J., Fan, T., Hu, J.: Least-squares images for edge-preserving smoothing. Comput. Visual Media 1(1), 27–35 (2015)

    Article  Google Scholar 

  18. Shao, H., Chen, S., Zhao, J., Cui, W., Yu, T.: Face recognition based on subset selection via metric learning on manifold. Front. Inf. Technol. Electron. Eng. 16(12), 1046–1058 (2015)

    Google Scholar 

  19. Oikawa, M.A., Dias, Z., de Rezende Rocha, A., Goldenstein, S.: Manifold learning and spectral clustering for image phylogeny forests. IEEE Trans. Inf. Forensics Secur. 11(1), 5–18 (2016)

    Article  Google Scholar 

  20. Blanz, V., Basso, C., Poggio, T., Vetter, T.: Reanimating faces in images and video. Comput. Graph. Forum 22(3), 641–650 (2003)

    Article  Google Scholar 

  21. Vlasic, D., Brand, M., Pfister, H., Popovic, J.: Face transfer with multilinear models. ACM Trans. Graph. (TOG) 24(3), 426–433 (2005)

    Article  Google Scholar 

  22. Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.: FaceWarehouse: a 3D facial expression database for visual computing. IEEE Trans. Visual Comput. Graph. 20(3), 413–425 (2014)

    Article  Google Scholar 

  23. Cao, C., Wu, H., Weng, Y., Shao, T., Zhou, K.: Real-time facial animation with image-based dynamic avatars. ACM Trans. Graph. (TOG) 35(4), 1–12 (2016)

    Article  Google Scholar 

  24. Saito, S., Li, T., Li, H.: Real-time facial segmentation and performance capture from RGB input. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 244–261. Springer, Cham (2016). doi:10.1007/978-3-319-46484-8_15

    Chapter  Google Scholar 

  25. Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194. ACM Press/Addison-Wesley Publishing Co., New York, USA (1999)

    Google Scholar 

  26. Lin, Y., Wang, S., Lin Q., Tang, F.: Face swapping under large pose variations: a 3D model based approach. In: IEEE International Conference on Multimedia and Expo (ICME), pp. 333–338. IEEE, Melbourne, VIC, Australia (2012)

    Google Scholar 

  27. Song, H., Lv, J., Liu, H., Zhao, Q.: A face replacement system based on 3D face model. In: Deng, Z., Li, H. (eds.) Proceedings of the 2015 Chinese Intelligent Automation Conference. LNEE, vol. 336, pp. 237–246. Springer, Heidelberg (2015). doi:10.1007/978-3-662-46469-4_25

    Chapter  Google Scholar 

  28. Lin, Y., Lin, Q., Tang, F., Wang, S.: Face replacement with large-pose differences. In: 20th ACM International Conference on Multimedia, pp. 1249–1250. ACM, Nara, Japan (2012)

    Google Scholar 

  29. Tran, A.T., Hassner, T., Masi, I., Medioni, G.: Regressing robust and discriminative 3D morphable models with a very deep neural network. arXiv preprint arXiv:1612.04904 (2017)

  30. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874. IEEE, Columbus, OH, USA (2014)

    Google Scholar 

  31. Huber, P., Hu, G., Tena, R., Mortazavian, P., Koppen W.P., Christmas, W., Ratsch, M., Kittler, J.: A multiresolution 3D Morphable Face Model and fitting framework. In: 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, pp. 1–8 (2016)

    Google Scholar 

  32. Sorkine, O.: Least-squares rigid motion using svd. Tech. Notes 120(3), 52 (2009)

    Google Scholar 

  33. Takayama, K., Schmidt, R., Singh, K., Igarashi, T., Boubekeur, T., Sorkine, O.: GeoBrush: interactive mesh geometry cloning. Comput. Graph. Forum 30(2), 613–622 (2011)

    Article  Google Scholar 

  34. Yu, Y., Zhou, K., Xu, D., Shi, X., Bao, H., Guo, B., Shum, H.-Y.: Mesh editing with poisson-based gradient field manipulation. ACM Trans. Graph. (TOG) 23(3), 644–651 (2004)

    Article  Google Scholar 

  35. Schmidt, R., Singh, K.: Drag, drop, and clone: an interactive interface for surface composition. Technical Report CSRG-611, Department of Computer Science, University of Toronto (2010)

    Google Scholar 

  36. Perez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. (TOG) 22(3), 313–318 (2003)

    Article  Google Scholar 

  37. Libigl. http://libigl.github.io/libigl/. Accessed 2016

  38. Zhao, J., Tang, M., Tong, R.: Mesh segmentation for parallel decompression on GPU. In: Hu, S.-M., Martin, R.R. (eds.) CVM 2012. LNCS, vol. 7633, pp. 83–90. Springer, Heidelberg (2012). doi:10.1007/978-3-642-34263-9_11

    Chapter  Google Scholar 

  39. Tang, X., Guo, J., Li, P., Lv, J.: A surgical simulation system for predicting facial soft tissue deformation. Comput. Visual Media 2(2), 163–171 (2016)

    Article  Google Scholar 

Download references

Acknowledgements

The research is supported in part by NSFC (61572424) and the People Programme (Marie Curie Actions) of the European Union’s Seventh Framework Programme FP7 (2007–2013) under REA grant agreement No. 612627-“AniNex”. Min Tang is supported in part by NSFC (61572423) and Zhejiang Provincial NSFC (LZ16F020003).

Author information

Authors and Affiliations

  1. State Key Lab of CAD&CG, Zhejiang University, Hangzhou, 310027, China

    Jiang Du, Dan Song, Ruofeng Tong & Min Tang

  2. The Third Affiliated Hospital, Zhejiang Chinese Medicine University, Hangzhou, China

    Yanjing Wu

Authors
  1. Jiang Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanjing Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dan Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruofeng Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Min Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ruofeng Tong .

Editor information

Editors and Affiliations

  1. Bournemouth University, Bournemouth, United Kingdom

    Jian Chang

  2. Bournemouth University, Bournemouth, United Kingdom

    Jian Jun Zhang

  3. University of Geneva , Carouge, Switzerland

    Nadia Magnenat Thalmann

  4. Tsinghua University, Beijing, China

    Shi-Min Hu

  5. Zhejiang University, Hangzhou, China

    Ruofeng Tong

  6. Institute of Software, Chinese Academy of Sciences, Beijing, China

    Wencheng Wang

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Du, J., Wu, Y., Song, D., Tong, R., Tang, M. (2017). Replacement of Facial Parts in Images. In: Chang, J., Zhang, J., Magnenat Thalmann, N., Hu, SM., Tong, R., Wang, W. (eds) Next Generation Computer Animation Techniques. AniNex 2017. Lecture Notes in Computer Science(), vol 10582. Springer, Cham. https://doi.org/10.1007/978-3-319-69487-0_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-69487-0_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-69486-3

  • Online ISBN: 978-3-319-69487-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation