Abstract
Neural Style Transfer enables synthesizing an image in the style of a reference image with convolutional neural networks. Photo-realistic style transfer aims to retain the photorealism in the synthesized image. Many of the state-of-the-art photo-realistic style transfer techniques uses networks like VGG as the backbone effectively making it computationally expensive and infeasible for deployment in systems with limited resources like mobile devices. In this work, it is shown that lightweight VGG like networks can be trained with knowledge distillation technique to achieve similar performance with ~35x lesser model parameters. In addition, a novel and efficient method is proposed to train neural networks to learn the image smoothing operator to enhance the photorealism in the stylized images. The proposed improvements, thus makes it feasible to achieve effective and fast on-device photo-realistic style transfer.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gatys, L., et al.: Image style transfer using convolutional neural networks. In: CVPR (2016)
Huang, X., et al.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV (2017)
Li, Y., et al.: Universal style transfer via feature transforms. In: NIPS (2017)
Luan, F., et al.: Deep photo style transfer. In: CVPR (2017)
Li, Y., et al.: A closed-form solution to photorealistic image stylization. In: ECCV (2018)
Li, X., et al.: learning linear transformations for fast arbitrary style transfer. In: CVPR (2019)
Hinton, G., et al.: Distilling the knowledge in a neural network. In: NIPS (2015)
Bucila, C., et al.: Model compression. In: SIGKDD (2006)
Ba, L.J., et al.: Do deep nets really need to be deep? In: NIPS (2014)
Porting Arbitrary Style Transfer to the Browser. https://magenta.tensorflow.org/blog/2018/12/20/style-transfer-js/
Simonyan, K., et al.: Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR, abs/1409.1556 (2014)
Chen, Q., et al.: Fast image processing with fully-convolutional networks. In: ICCV (2017)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., W, Max (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
He, K., et al.: Guided Image Filtering. In: TPAMI (2013)
Lin, T.Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Sen, M., Babu, M.N., Mittar, R., Gupta, D., Chakraborty, P. (2021). Lightweight Photo-Realistic Style Transfer for Mobile Devices. In: Singh, S.K., Roy, P., Raman, B., Nagabhushan, P. (eds) Computer Vision and Image Processing. CVIP 2020. Communications in Computer and Information Science, vol 1377. Springer, Singapore. https://doi.org/10.1007/978-981-16-1092-9_19
Download citation
DOI: https://doi.org/10.1007/978-981-16-1092-9_19
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-1091-2
Online ISBN: 978-981-16-1092-9
eBook Packages: Computer ScienceComputer Science (R0)