Skip to main content
SpringerLink
Log in

Caricature synthesis with feature deviation matching under example-based framework

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Example-based caricature synthesis techniques have been attracting large attentions for being able to generate attractive caricatures of various styles. This paper proposes a new example-based caricature synthesis system using a feature deviation matching method as a cross-modal distance metric. It employs the deviation values from average features across different feature spaces rather than the values of features themselves to search for similar components from caricature examples directly. Compared with traditional example-based systems, the proposed system can generate various styles of caricatures without requiring paired photograph–caricature example databases. The newly designed features can effectively capture visual characteristics of the hairstyles and facial components in input portrait images. In addition, this system can control the exaggeration of individual facial components and provide several similarity-based candidates to satisfy users’ different preferences. Experiments are conducted to prove the above ideas.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  1. Sadimon, S.B., Sunar, M.S., Mohamad, D., Haron, H.: Computer generated caricature: a survey. In: Proceedings of the International Conference on Cyberworlds, pp. 383–390 (2010)

  2. Chen, H., Zheng, N.N., Liang, L., Li, Y., Xu, Y.Q., Shum, H.Y.: PicToon: a personalized image-based cartoon system. In: Proceedings of the tenth ACM international conference on Multimedia, pp. 171–178 (2002)

  3. Wang, N.N., Tao, D.C., Gao, X.B., Li, X.L., Li, J.: Transductive face sketch-photo synthesis. IEEE Trans. Neural Netw. Learn. Syst. 24(9), 1364–1376 (2013)

    Article  Google Scholar 

  4. Min, F., Suo, J.L., Zhu, S.C., Sang, N.: An automatic portrait system based on and-or graph representation. In: International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR), pp. 184–197 (2007)

  5. Xu, Z.J., Chen, H., Zhu, S.C., Luo, J.B.: A hierarchical compositional model for face representation and sketching. IEEE Trans. Pattern Anal. Mach. Intell. 30(6), 955–969 (2008)

    Article  Google Scholar 

  6. Gooch, B., Reinhard, E., Gooch, A.: Human facial illustrations: creation and psychophysical evaluation. ACM Trans. Graph. 23(1), 27–44 (2004)

    Article  Google Scholar 

  7. Wang, X.G., Tang, X.O.: Face photo-sketch synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 1955–1967 (2009)

    Article  Google Scholar 

  8. Song, Y.B., Bao, L.C., Yang, Q.X., Yang, M.H.: Real-time exemplar based face sketch synthesis. Comput. Vis. ECCV 2014, 800–813 (2014)

    Google Scholar 

  9. Zhang, D.Y., Lin, L., Chen, T.S., Wu, X., Tan, W.W., Izquierdo, E.: Content-adaptive sketch portrait generation by decompositional representation learning. IEEE Trans. Image Process. 26(1), 328–339 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  10. Chen, H., Liu, Z.Q., Rose, C., Xu, Y.Q., Shum, H.Y., Salesin, D.: Example-based composite sketching of human portraits. In: Proceedings of the 3rd International Symposium on Non-photorealistic Animation and Rendering, pp. 7–9 (2004)

  11. Chen, H., Xu, Y.Q., Shum, H.Y., Zhu, S.C., Zheng, N.N.: Example-based facial sketch generation with non-parametric sampling. Proc. Int. Conf. Comput. Vis. 2, 433–438 (2001)

    Google Scholar 

  12. Chen, W.J., Yu, H.C., Zhang, J.J: Example based caricature synthesis. Adv. Comput. Sci. Eng. 5(1) (2010)

  13. Liang, L., Chen, H., Xu, Y.Q., Shum, H.Y.: Example-based caricature generation with exaggeration. In: Computer Graphics and Applications, Pacific Conference, pp. 386–393 (2002)

  14. Yang, W., Tajima, K., Xu, J.Y., Toyoura, M., Mao, X.X.: Example-based automatic caricature generation. In: Cyberworlds, pp. 237–244 (2014)

  15. Yu, L.F., Yeung, S.K., Terzopoulos, D., Chan, T.F.: DressUp! Outfit synthesis through automatic optimization. ACM Trans. Graph. 31(6), 134:1–134:14 (2012)

    Article  Google Scholar 

  16. Kalogerakis, E., Chaudhuri, S., Koller, D., Koltun, V.: A probabilistic model for component-based shape synthesis. ACM Trans. Graph. 31(4), 55:1–55:11 (2012)

    Article  Google Scholar 

  17. Yang, W., Toyoura, M., Xu, J.J., Ohnuma, F., Mao, X.Y.: Example-based caricature generation with exaggeration control. Vis. Comput. 32(3), 383–392 (2016)

    Article  Google Scholar 

  18. Zhang, Y., Dong, W.M., Ma, C.Y., Mei, X., Li, K., Huang, F.Y., Hu, B.G., Deuusen, O.: Data-driven synthesis of cartoon faces using different styles. IEEE Trans. Image Process. 26(1), 464–478 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  19. Shih, Y.C., Paris, S., Barnes, C., Freeman, W.T., Durand, Frédo: Style transfer for headshot portraits. ACM Trans. Graph. 33(4), 148 (2014)

    Article  Google Scholar 

  20. Selim, A., Elgharib, M., Doyle, L.: Painting style transfer for head portraits using convolutional neural networks. ACM Trans. Graph. 35(4), 129 (2016)

    Article  Google Scholar 

  21. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR (2016)

  22. Liao, J., Yao, Y., Yuan, L., Hua, G., Kang, S.B.: Visual Attribute Transfer Through Deep Image Analogy (2017). arXiv:1705.01088 [cs.CV]

  23. Fišer, J., Jamriška, O., Simons, D., Shechtman, E., Lu, J.W., Asente, P., Luká, M., Sýkora, D.: Example-based synthesis of stylized facial animations. ACM Trans. Graph. 36(4), 155 (2017)

    Article  Google Scholar 

  24. Taigman, Y., Polyak, A., Wolf, L.: Unsupervised cross-domain image generation. In: ICLR (2017)

  25. Harmon, L.D.: The recognition of faces. Sci. Am. 229(5), 71–82 (1973)

    Article  Google Scholar 

  26. Brennan, S.E.: Caricature generator: the dynamic exaggeration of faces by computer. Leonardo 18(3), 170–178 (1985)

    Article  Google Scholar 

  27. Koshimizu, H., Tominaga, M., Fujiwara, T., Murakami, K.: On KANSEI facial image processing for computerized facial caricaturing system Picasso. In: IEEE International Conference on Systems, pp. 294–299 (1999)

  28. Mo, Z.Y., Lewis, J.P., Neumann, U.: Improved automatic caricature by feature normalization and exaggeration. In: ACM SIGGRAPH Sketches, p. 57 (2004)

  29. Xu, J.Y., Yang, W., Mao, X.Y., Toyoura, M., Jin, X.G.: A study on perceived similarity between photograph and shape exaggerated caricature. In: International Conference on Cyberworlds, pp. 213–220 (2014)

  30. Cosker, D., Roy, S., Rosin, P.L., Marshall, D.: Re-mapping animation parameters between multiple types of facial model. In: International Conference on Computer Vision/Computer Graphics Collaboration Techniques, pp. 365–276 (2007)

  31. Hotelling, H.: Relations between two sets of variates. Biometrika 28(3/4), 321–377 (1936)

    Article  MATH  Google Scholar 

  32. Rasiwasia, N., Pereira, J.C., Coviello, E., Doyle, G., Lanckriet, G.R.G., Levy, R., Vasconcelos, N.: A new approach to cross-modal multimedia retrieval. In: ACM International Conference on Multimedia, pp. 251–260 (2010)

  33. Cao, Y., Long, M.S., Wang, J.M., Liu, S.C.: Collective deep quantization for efficient cross-modal retrieval. In: IEEE International Symposium on Multimedia, pp. 3974–3980 (2017)

  34. Zhong, C.L., Yu, Y., Tang, S.H., Satoh, S., Xing, K.: Deep multi-label hashing for large-scale visual search based on semantic graph. In: Spring International Publishing, pp. 169–184 (2017)

  35. Yan, F., Mikolajczyk, K.: Deep correlation for matching images and text. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3441–3450 (2015)

  36. Andrew, G., Arora, R., Bilmes, J., Livescu, K.: Deep canonical correlation analysis. In: International Conference on International Conference on Machine Learning, pp. –1247–1255 (2013)

  37. Yu, Y., Tang, S.H., Raposo, F., Chen, L.: Deep Cross-modal Correlation Learning for Audio and Lyrics in Music Retrieval (2017). arXiv:1711.08976 [cs.IR]

  38. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Comput. Vis. Image Underst. 61(1), 38–59 (1995)

    Article  Google Scholar 

  39. Li, H.L., Yang, W., Sun, H.C., Toyoura, M., Mao, X.Y.: Example-based caricature synthesis via feature deviation matching. In: CGI’16 Proceedings of the 33rd Computer Graphics International, pp. 81–84 (2016)

  40. Yacoob, Y., Davis, L.S.: Detection and analysis of hair. IEEE Trans. Pattern Anal. Mach. Intell. 28(7), 1164–1169 (2006)

    Article  Google Scholar 

  41. Luc, V., Pierre, S.: Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Trans. Pattern Anal. Mach. Intell. 13(6), 583–598 (1991)

    Article  Google Scholar 

  42. Bookstein, F.L.: Principal warps: thin-plate splines and the decomposition of deformations. IEEE Trans. Pattern Anal. Mach. Intell. 11(6), 567–585 (1989)

    Article  MATH  Google Scholar 

  43. Li, H.L., Toyoura, M., Shimizu, K., Yang, W., Mao, X.Y.: Retrieval of clothing images based on relevance feedback with focus on collar designs. Vis. Comput. 32(10), 1351–1363 (2016)

    Article  Google Scholar 

  44. Yang, W., Toyoura, M., Mao, X.Y.: Hairstyle suggestion using statistical learning. In: International Conference on Multimedia Modeling, pp. 277–287 (2012)

  45. Sunhem, W., Pasupa, K.: An approach to face shape classification for hairstyle recommendation. In: Eighth International Conference on Advanced Computational Intelligence, pp. 390–394 (2016)

Download references

Acknowledgements

This study was funded by JSPS Grants-in-Aid for Scientific Research (Grant Nos. 26560006 and 26240015) and (Grant No. KAKENHI 17H00737).

Author information

Authors and Affiliations

  1. University of Yamanashi, 4-3-11, Kofu, Yamanashi, 400-8511, Japan

    Honglin Li, Masahiro Toyoura & Xiaoyang Mao

Authors
  1. Honglin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Masahiro Toyoura
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoyang Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoyang Mao.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Toyoura, M. & Mao, X. Caricature synthesis with feature deviation matching under example-based framework. Vis Comput 35, 653–666 (2019). https://doi.org/10.1007/s00371-018-1495-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-018-1495-9

Keywords

Search

Navigation