Abstract
Face sketch synthesis has a wide range of applications in both digital entertainment and law enforcement. State-of-the-art examplar-based methods typically exploit a Probabilistic Graphical Model (PGM) to represent the joint probability distribution over all of the patches selected from a set of training data. However, these methods suffer from two main shortcomings: (1) most of these methods capture the evidence between patches in pixel-level, which lead to inaccurate parameter estimation under bad environment conditions such as light variations and clutter backgrounds; (2) the assumption that a photo patch and its corresponding sketch patch share similar geometric manifold structure is not rigorous. It has shown that deep convolutional neural network (CNN) has outstanding performance in learning to extract high-level feature representation. Therefore, we extract uniform deep patch representations of test photo patches and training sketch patches from a specially designed CNN model to replace pixel intensity, and directly match between them, which can help select better candidate patches from training data as well as improve parameter learning process. In this way, we investigate a novel face sketch synthesis method called DPGM that combines generative PGM and discriminative deep patch representation, which can jointly model the distribution over the parameters for deep patch representation and the distribution over the parameters for sketch patch reconstruction. Then, we apply an alternating iterative optimization strategy to simultaneously optimize two kinds of parameters. Therefore, both the representation capability of deep patch representation and the reconstruction ability of sketch patches can be boosted. Eventually, high quality reconstructed sketches which is robust against light variations and clutter backgrounds can be obtained. Extensive experiments on several benchmark datasets demonstrate that our method can achieve superior performance than other state-of-the-art methods, especially under the case of bad light conditions or clutter backgrounds.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Beis, J. S., & Lowe, D. G. (1997). Shape indexing using approximate nearest-neighbour search in high-dimensional spaces. In Proceedings of IEEE computer society conference on computer vision and pattern recognition (pp 1000–1006).
Chang, L., Zhou, M., Deng, X., Wu, Z., & Han, Y. (2011). Face sketch synthesis via multivariate output regression. In International conference on human-computer interaction (pp 555–561).
Chen, L., Liao, H., Ko, M., Lin, J., & Yu, G. (2000). A new lda-based face recognition system which can solve the small sample size problem. Pattern Recognition, 33(10), 1713–1726.
Gao, X., Wang, N., Tao, D., & Li, X. (2012). Face sketch-photo synthesis and retrieval using sparse representation. IEEE Transactions on Circuits and Systems for Video Technology, 22(8), 1213–1226.
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016). Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp 2414–2423).
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, Z., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp 2672–2680).
Isola, P., Zhu, J., Zhou, T., & Efros, A. (2016). Image-to-image translation with conditional adversarial networks. arXiv preprint p arXiv:1611.07004.
Jegou, H., Douze, M., & Schmid, C. (2010). Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1), 117–128.
Johnson, J., Alahi, A., & Li, F. (2016). Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision (pp 694–711).
Li, C. (2016). Combining markov random fields and convolutional neural networks for image synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp 2479–2486).
Liu, Q., Tang, X., Jin, H., Lu, H., & D Ma, S. (2005). A nonlinear approach for face sketch synthesis and recognition. In Proceedings of IEEE conference on computer vision and pattern recognition (pp 1005–1010).
Lu, C., Xu, L., & Jia, J. (2012). Combining sketch and tone for pencil drawing production. In International symposium on non-photorealistic animation and rendering.
Mao, X., Nagasaka, Y., & Imamiya, A. (2001). Automatic generation of pencil drawing from 2d images using line integral convolution. In CAD/Graphics (pp 240–248).
Martinez, A., Benavente, R. (1998). The AR face database. CVC Technical Report #24.
Meer, P., Mintz, D., Rosenfeld, A., & Kim, D. (1991). Robust regression methods for computer vision: A review. International Journal of Computer Vision, 6(1), 59–70.
Messer, K., Matas, J., Kittler, J., Luettin, J., & Maitre, G. (1999). Xm2vtsdb: the extended m2vts database. In Proceedings of international conference on audio- and video-based biometric person authentication, (pp 72–77).
Paszke, A., Gross, S., & Chintala, S. (2017). Pytorch deep learning framework. Web page, http://pytorch.org/.
Phillips, P., Moon, H., Rizvi, S., & Rauss, P. (2000). The feret evaluation methodology for face recognition algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(10), 1090–1104.
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. CoRR. (abs/1409.1556).
Song, Y., Bao, L., Yang, Q., & M. Y. (2014) Real-time exemplar-based face sketch synthesis. In Proceedings of Eureopean conference on computer vision (pp 800–813).
Tang, X., Wang, X. (2002). Face photo recognition using sketch. In Proceedings of IEEE international conference on image processing (pp 257–260).
Tang, X., Wang, X. (2003) Face sketch synthesis and recognition. In Proceedings of IEEE international conference on computer vision (pp 687–694).
Tang, X., & Wang, X. (2004). Face sketch recognition. IEEE Transactions on Circuits and Systems for Video Technology, 14(1), 1–7.
Wang, N., Li, J., Tao, D., Li, X., & Gao, X. (2013). Heterogeneous image transformation. Pattern Recognition Letters, 34(1), 77–84.
Wang, N., Tao, D., Gao, X., Li, X., & Li, J. (2014). A comprehensive survey to face hallucination. International Journal of Computer Vision, 106(1), 9–30.
Wang, N., Gao, X., & Li, J. (2017a). Random sampling and locality constraint for fast face sketch synthesis. arXiv preprint arXiv:1701.01911.
Wang, N., Gao, X., Sun, L., & Li, J. (2017b). Bayesian face sketch synthesis. In IEEE transactions on image processing (pp 1–11).
Wang, X., & Tang, X. (2009). Face photo-sketch synthesis and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(11), 1955–1967.
Wang, Z., Bovik, A., Sheikh, H., & Simoncelli, E. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.
Yang, M., Zhang, L., Feng, X., & Zhang, D. (2014). Sparse representation based fisher discrimination dictionary learning for image classification. International Journal of Computer Vision, 109(3), 209–232.
Zhang, J., Wang, N., Gao, X., Tao, D., & Li, X. (2011a) Face sketch-photo synthesis based on support vector regression. In IEEE International Conference on Image Processing (pp 1125–1128)
Zhang, L., Lin, L., Wu, X., Ding, S., & Zhang, L. (2015) End-to-end photo-sketch generation via fully convolutional representation learning. In Proceedings of the 5th ACM on international conference on multimedia retrieval (pp 627–634).
Zhang, W., Wang, X., & Tang, X. (2011b) Coupled information-theoretic encoding for face photo-sketch recognition. In Proceedings of IEEE conference on computer vision and pattern recognition (pp 513–520).
Zhou, H., Kuang, Z., & Wong, K. (2012) Markov weight fields for face sketch synthesis. In Proceedings of IEEE conference on computer vision and pattern recognition (pp 1091–1097).
Zhu, M., Wang, N. (2016) A simple and fast method for face sketch synthesis. In International conference on internet multimedia computing and service (pp 168–171).
Zhu, M., Wang, N., Gao, X., & Li, J. (2017) Deep graphical feature learning for face sketch synthesis. In Proceedings of the twenty-sixth international joint conference on artificial intelligence, IJCAI-17 (pp 3574–3580).
Zhu, M., Li, J., Wang, N., & Gao, X. (2018). A deep collaborative framework for face photo-sketch synthesis. IEEE Transactions on Neural Networks and Learning Systems Early Access,. https://doi.org/10.1109/TNNLS.2018.2872675.
Acknowledgements
This work was supported in part by the National Key Research and Development Program of China under Grant 2018AAA0103202; in part by the National Natural Science Foundation of China under Grant Grants 62036007, 61922066, 61876142, 61772402, and 62050175; in part by the Xidian University Intellifusion Joint Innovation Laboratory of Articial Intelligence; in part by the Fundamental Research Funds for the Central Universities.
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Recommended by: Xiaoming Liu, Chen Change Loy, Rama Chellappa, Tae-Kyun Kim, Tae-Kyun Kim.
Rights and permissions
About this article
Cite this article
Zhu, M., Li, J., Wang, N. et al. Learning Deep Patch representation for Probabilistic Graphical Model-Based Face Sketch Synthesis . Int J Comput Vis 129, 1820–1836 (2021). https://doi.org/10.1007/s11263-021-01442-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-021-01442-2