Abstract
In this paper, we present a novel approach for transferring shape and expression of a face in image to that of another, regardless of variance between the two faces in illumination, color, texture, resolution and even some mild occlusion. We first use a face alignment algorithm to locate accurate facial landmark points for both original face and target face, then align them with a global similarity transformation to eliminate their inconsistency in pose, size and position. Finally, we use our non-rigid image deformation method to deform the original face by fitting a map function for each of its pixel point according to the two sets of facial landmark points. Our method can be full-automatic or semi-automatic for conveniently tuning a better result by combining a face alignment algorithm and a non-rigid image deformation method. Experiment results show that our method can produce realistic, natural and artifact-less facial shape and expression transfer. We also discuss the limitation and potential of our proposed method.
The authors gratefully acknowledge the financial supports from the National Natural Science Foundation of China under Grant Nos. 41501505, 61502354 and the Scientific Research Project of Education Department of Hubei Province under Grant No. Q20181508.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Alexa, M., Cohen-Or, D., Levin, D.: As-rigid-as-possible shape interpolation. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 157–164 (2000)
Aronszajn, N.: Theory of reproducing kernels. Trans. Am. Math. Soc. 68(3), 337–404 (1950)
Beier, T., Neely, S.: Feature-based image metamorphosis. ACM SIGGRAPH Comput. Graph. 26(2), 35–42 (1992)
Bookstein, F.L.: Principal warps: thin-plate splines and the decomposition of deformations. IEEE Trans. Pattern Anal. Mach. Intell. 11(6), 567–585 (2002)
Gangnet, M., Blake, A.: Poisson image editing. In: ACM SIGGRAPH, pp. 313–318 (2003)
Garrido, P., Valgaerts, L., Rehmsen, O., Thormaehlen, T., Perez, P., Theobalt, C.: Automatic face reenactment. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4217–4224 (2014)
Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Computer Vision and Pattern Recognition. pp. 1867–1874 (2014)
Levin, D.: The approximation power of moving least-squares. Math. Comput. 67(224), 1517–1531 (1998)
Liu, L., Liu, L., Nie, X., Feng, J., Yan, S., Yan, S.: A live face swapper. In: ACM on Multimedia Conference, pp. 691–692 (2016)
Ma, J., Zhao, J., Tian, J., Bai, X., Tu, Z.: Regularized vector field learning with sparse approximation for mismatch removal. Pattern Recognit. 46(12), 3519–3532 (2013)
Ma, J., Zhao, J., Tian, J., Yuille, A.L., Tu, Z.: Robust point matching via vector field consensus. IEEE Trans. Image Process. 23(4), 1706–1721 (2014)
Ma, J., Zhao, J., Guo, H., Jiang, J., Zhou, H., Gao, Y.: Locality preserving matching. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 4492–4498. AAAI Press (2017)
Ma, J., Zhao, J., Jiang, J., Zhou, H.: Non-rigid point set registration with robust transformation estimation under manifold regularization. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4218–4224 (2017)
Ma, J., Zhao, J., Tian, J.: Nonrigid image deformation using moving regularized least squares. IEEE Signal Process. Lett. 20(10), 988–991 (2013)
Ma, J., Zhao, J., Tian, J., Tu, Z., Yuille, A.L.: Robust estimation of nonrigid transformation for point set registration. In: Proceedings IEEE Conference Computer Vision Pattern Recognition, pp. 2147–2154 (2013)
Ma, J., Zhao, J., Tian, J., Yuille, A.L., Tu, Z.: Robust point matching via vector field consensus. IEEE Trans. Image Process. 23(4), 1706–1721 (2014)
Maccracken, R., Joy, K.I.: Free-form deformations with lattices of arbitrary topology. In: Conference on Computer Graphics and Interactive Techniques, pp. 181–188 (1996)
Min, F., Sang, N., Wang, Z.: Automatic face replacement in video based on 2D morphable model. In: International Conference on Pattern Recognition, pp. 2250–2253 (2010)
Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., Salesin, D.H.: Synthesizing realistic facial expressions from photographs (1998)
Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 fps via regressing local binary features. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1685–1692 (2014)
Schaefer, S., Mcphail, T., Warren, J.: Image deformation using moving least squares. ACM Trans. Graph. 25(3), 533–540 (2006)
Shen, S., Yamasaki, T., Aizawa, K., Sugahara, T.: Data-driven geometric face image smilization featuring moving least square based deformation. In: IEEE Third International Conference on Multimedia Big Data, pp. 220–225 (2017)
Xiao, S., Yan, S., Kassim, A.A.: Facial landmark detection via progressive initialization. In: IEEE International Conference on Computer Vision Workshop, pp. 986–993 (2015)
Zhang, Z., Luo, P., Loy, C.C., Tang, X.: Facial landmark detection by deep multi-task learning. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 94–108. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_7
Zhou, H., Kuang, Y., Yu, Z., Ren, S., Dai, A., Zhang, Y., Lu, T., Ma, J.: Non-rigid image deformation algorithm based on MRLS-TPS. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 2269–2273. IEEE (2017)
Zhou, H., Ma, J., Yang, C., Sun, S., Liu, R., Zhao, J.: Nonrigid feature matching for remote sensing images via probabilistic inference with global and local regularizations. IEEE Geosci. Remote Sens. Lett. 13(3), 374–378 (2016)
Zhou, H., Ma, J., Zhang, Y., Yu, Z., Ren, S., Chen, D.: Feature guided non-rigid image/surface deformation via moving least squares with manifold regularization. In: 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 1063–1068. IEEE (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, H. et al. (2018). Facial Shape and Expression Transfer via Non-rigid Image Deformation. In: Vaidya, J., Li, J. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2018. Lecture Notes in Computer Science(), vol 11336. Springer, Cham. https://doi.org/10.1007/978-3-030-05057-3_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-05057-3_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-05056-6
Online ISBN: 978-3-030-05057-3
eBook Packages: Computer ScienceComputer Science (R0)