Skip to main content
SpringerLink
Log in

A Consistency Enhanced Deep Lmser Network for Face Sketch Synthesis

  • Conference paper
  • First Online:
PRICAI 2021: Trends in Artificial Intelligence (PRICAI 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13031))

Included in the following conference series:

Abstract

Existing face sketch synthesis methods extend conditional generative adversarial network framework with promising performance. However, they usually pre-train on additional large-scale datasets, and the performance is still not satisfied. To tackle the issues, we develop a deep bidirectional network based on the least mean square error reconstruction (Lmser) self-organizing network, which is a further development of autoencoder by folding along the central hidden layer. Such folding makes the neurons on the paired layers between encoder and decoder merge into one. We model the photo-to-sketch mapping by an Lmser network and build a sketch-to-photo mapping by a complement Lmser sharing the same structure. The bidirectional mappings form an alternation system. We devise a supervised alternating consistency for the system, by minimizing the deviation between the alternatively mapped pattern and the ground-truth. Enhanced by the consistency constraints along the bidirectional paths, the model achieve a significant improvement in terms of Fréchet Inception Distance (FID). Experiments demonstrate the effectiveness of our method in comparison with state-of-the-art methods, and reduce the FID from 34.1 to 28.7 on the CUFS dataset and from 18.2 to 11.5 on the CUFSF dataset.

Supported by National Science and Technology Innovation 2030 Major Project (2018AAA0100700) of the Ministry of Science and Technology of China and Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: Stylebank: an explicit representation for neural image style transfer. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2770–2779 (2017)

    Google Scholar 

  2. Chen, L., Liao, H., Ko, M., Lin, J., Yu, G.: A new lda-based face recognition system which can solve the small sample size problem. Pattern Recogn. 33(10), 1713–1726 (2000)

    Article  Google Scholar 

  3. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680 (2014)

    Google Scholar 

  4. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 30, pp. 6626–6637 (2017)

    Google Scholar 

  5. Huang, W., Tu, S., Xu, L.: Revisit lmser from a deep learning perspective. In: Intelligence Science and Big Data Engineering. Big Data and Machine Learning, pp. 197–208 (2019)

    Google Scholar 

  6. Huang, W., Tu, S., Xu, L.: Deep CNN based Lmser and strengths of two built-in dualities. Neural Process. Lett. 8, 108418–108428 (2020)

    Google Scholar 

  7. Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5967–5976 (2017)

    Google Scholar 

  8. Johson, J., Alahi, A., Li, F.F.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711 (2016)

    Google Scholar 

  9. Karacan, L., Erdem, E., Erdem, A.: Structure-preserving image smoothing via region covariances 32(6), 176 (2013)

    Google Scholar 

  10. Li, P., Tu, S., Xu, L.: GAN flexible Lmser for super-resolution. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 756–764 (2019)

    Google Scholar 

  11. Liu, M., Tuzel, O.: Coupled generative adversarial networks. In: Advances in Neural Information Processing Systems, vol. 29, pp. 469–477 (2016)

    Google Scholar 

  12. Liu, Q., Tang, X., Jin, H., Lu, H., Ma, S.: A nonlinear approach for face sketch synthesis and recognition. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 1005–1010 (2005)

    Google Scholar 

  13. Liu, S., Yang, J., Huang, C., Yang, M.: Multi-objective convolutional learning for face labeling. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3451–3459 (2015)

    Google Scholar 

  14. Lucic, M., Kurach, K., Michalski, M., Gelly, S., Bousquet, O.: Are gans created equal? a large-scale study. In: Advances in Neural Information Processing Systems, vol. 31, pp. 700–709 (2018)

    Google Scholar 

  15. Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., Frey, B.: Adversarial autoencoders. arXiv preprint arXiv:1511.05644 (2015)

  16. Mao, X., Li, Q., Xie, H., Lau, R.Y.K., Wang, Z., Smolley, S.P.: Least squares generative adversarial networks. In: IEEE International Conference on Computer Vision, pp. 2813–2821 (2017)

    Google Scholar 

  17. Martinez, A., Benavente, R.: The AR face database. CVC Technical Report #24 (1998)

    Google Scholar 

  18. Messer, K., Matas, J., Kittler, J., Luettin, J., Maitre, G.: XM2VTSDB: The extended M2VTS database. In: Second International Conference on Audio and Video-based Biometric Person Authentication, pp. 72–77 (1999)

    Google Scholar 

  19. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  20. Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  21. Song, Y., Bao, L., Yang, Q., Yang, M.: Real-time exemplar-based face sketch synthesis. In: European Conference on Computer Vision, pp. 800–813 (2014)

    Google Scholar 

  22. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)

    Google Scholar 

  23. Tang, X., Wang, X.: Face photo recognition using sketch. In: Proceedings of the International Conference on Image Processing, vol. 1, pp. I-I (2002)

    Google Scholar 

  24. Tang, X., Wang, X.: Face sketch synthesis and recognition. In: Proceedings Ninth IEEE International Conference on Computer Vision, vol. 1, pp. 687–694 (2003)

    Google Scholar 

  25. Wang, N., Gao, X., Li, J.: Random sampling for fast face sketch synthesis. Pattern Recogn. 76, 215–227 (2018)

    Article  Google Scholar 

  26. Wang, N., Gao, X., Li, J., Song, B., Li, Z.: Evaluation on synthesized face sketches. Neurocomputing 214, 991–1000 (2016)

    Article  Google Scholar 

  27. Wang, N., Zha, W., Li, J., Gao, X.: Back projection: an effective postprocessing method for GAN-based face sketch synthesis. Pattern Recogn. Lett. 107, 59–65 (2017)

    Article  Google Scholar 

  28. Wang, X., Tang, X.: Face photo-sketch synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 1955–1967 (2009)

    Article  MathSciNet  Google Scholar 

  29. Xu, L.: Least mse reconstruction by self-organization. i. multi-layer neural-nets. In: Proceedings 1991 IEEE International Joint Conference on Neural Networks, vol. 3, pp. 2362–2367 (1991)

    Google Scholar 

  30. Xu, L.: Least mean square error reconstruction principle for self-organizing neural-nets. Neural Netw. 6(5), 627–648 (1993)

    Article  Google Scholar 

  31. Xu, L.: An overview and perspectives on bidirectional intelligence: Lmser duality, double IA harmony, and causal computation. IEEE/CAA J. Automatica Sinica 6(4), 865–893 (2019)

    Article  MathSciNet  Google Scholar 

  32. Yang, H., Tu, S.: Glmser: a gan-lmser network for image-to-image translation. In: IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), pp. 582–589 (2019)

    Google Scholar 

  33. Yu, J., et al.: Toward realistic face photo-sketch synthesis via composition-aided gans. IEEE Transactions on Cybernetics, pp. 1–13 (2020)

    Google Scholar 

  34. Zhang, D., Lin, L., Chen, T., Wu, X., Tan, W., Izquierdo, E.: Content-adaptive sketch portrait generation by decompositional representation learning. IEEE Trans. Image Process. 26(1), 328–339 (2017)

    Article  MathSciNet  Google Scholar 

  35. Zhang, L., Lin, L., Wu, X., Ding, S., Zhang, L.: End-to-end photo-sketch generation via fully convolutional representation learning. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pp. 627–634 (2015)

    Google Scholar 

  36. Zhang, L., Zhang, L., Mou, X., Zhang, D.: FSIM: a feature similarity index for image quality assessment. IEEE Trans. Image Process. 20(8), 2378–2386 (2011)

    Article  MathSciNet  Google Scholar 

  37. Zhang, M., Li, J., Wang, N., Gao, X.: Compositional model-based sketch generator in facial entertainment. IEEE Trans. Syst. Man Cybern. 48(3), 904–915 (2018)

    Google Scholar 

  38. Zhang, W., Wang, X., Tang, X.: Coupled information-theoretic encoding for face photo-sketch recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 513–520 (2011)

    Google Scholar 

  39. Zhu, J., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE International Conference on Computer Vision, pp. 2242–2251 (2017)

    Google Scholar 

  40. Zhu, M., Wang, N., Gao, X., Li, J.: Deep graphical feature learning for face sketch synthesis. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pp. 3574–3580 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

    Qingjie Sheng, Shikui Tu & Lei Xu

Authors
  1. Qingjie Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shikui Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shikui Tu .

Editor information

Editors and Affiliations

  1. MIMOS Berhad, Kuala Lumpur, Malaysia

    Duc Nghia Pham

  2. Sirindhorn International Institute of Science and Technology, Thammasat University, Mueang Pathum Thani, Thailand

    Thanaruk Theeramunkong

  3. Data61, CSIRO, Brisbane, QLD, Australia

    Guido Governatori

  4. Department of Philosophy, Tsinghua University, Beijing, China

    Fenrong Liu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sheng, Q., Tu, S., Xu, L. (2021). A Consistency Enhanced Deep Lmser Network for Face Sketch Synthesis. In: Pham, D.N., Theeramunkong, T., Governatori, G., Liu, F. (eds) PRICAI 2021: Trends in Artificial Intelligence. PRICAI 2021. Lecture Notes in Computer Science(), vol 13031. Springer, Cham. https://doi.org/10.1007/978-3-030-89188-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-89188-6_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-89187-9

  • Online ISBN: 978-3-030-89188-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation