Abstract
Video stream compression, using lossy algorithms, is performed to reduce the bandwidth required for transmission. To improve the video quality, either for human view or for automatic video analysis, videos are post-processed to eliminate the introduced compression artifacts. Generative Adversarial Network have been shown to obtain extremely high quality results in image enhancement tasks; however, to obtain top quality results high capacity large generators are usually employed, resulting in high computational costs and processing time. In this paper we present an architecture that can be used to reduce the cost of generators, paving a way towards real-time frame enhancement with GANs.
With the proposed approach, enhanced images appear natural and pleasant to the eye. Locally high frequency patterns often differ from the raw uncompressed images. A possible application is to improve video conferencing, or live streaming. In these cases there is no original uncompressed video stream available. Therefore, we report results using popular no-reference metrics showing high naturalness and quality even for efficient networks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agustsson, E., Timofte, R.: NTIRE 2017 challenge on single image super-resolution: dataset and study. In: Proceedings of IEEE CVPR Workshops (2017)
Bruna, J., Sprechmann, P., LeCun, Y.: Super-resolution with deep convolutional sufficient statistics. CoRR abs/1511.05666 (2015)
Cavigelli, L., Hager, P., Benini, L.: CAS-CNN: a deep convolutional neural network for image compression artifact suppression. In: Proceedings of IJCNN (2017)
Chu, M., Xie, Y., Leal-Taixé, L., Thuerey, N.: Temporally coherent GANs for video super-resolution (TecoGAN). arXiv preprint arXiv:1811.09393 (2018)
Dar, Y., Bruckstein, A.M., Elad, M., Giryes, R.: Postprocessing of compressed images via sequential denoising. IEEE Trans. Image Process. 25(7), 3044–3058 (2016)
Dong, C., Deng, Y., Change Loy, C., Tang, X.: Compression artifacts reduction by a deep convolutional network. In: Proceedings of ICCV (2015)
Dosovitskiy, A., Brox, T.: Generating images with perceptual similarity metrics based on deep networks. In: Proceedings of NIPS (2016)
Foi, A., Katkovnik, V., Egiazarian, K.: Pointwise shape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images. IEEE Trans. Image Process. 16(5), 1395–1411 (2007)
Galteri, L., Seidenari, L., Bertini, M., Del Bimbo, A.: Deep universal generative adversarial compression artifact removal. IEEE Trans. Multimed. 21(8), 2131–2145 (2019)
Galteri, L., Seidenari, L., Bertini, M., Del Bimbo, A.: Deep generative adversarial compression artifact removal. In: Proceedings of ICCV (2017)
Gatys, L.A., Ecker, A.S., Bethge, M.: Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks. CoRR abs/1505.07376 (2015)
Goodfellow, I., et al.: Generative adversarial nets. In: Proceedings of NIPS (2014)
He, X., Hu, Q., Zhang, X., Zhang, C., Lin, W., Han, X.: Enhancing HEVC compressed videos with a partition-masked convolutional neural network. In: Proceedings of ICIP (2018)
Jakhetiya, V., Lin, W., Jaiswal, S.P., Guntuku, S.C., Au, O.C.: Maximum a posterior and perceptually motivated reconstruction algorithm: a generic framework. IEEE Trans. Multimed. 19(1), 93–106 (2017)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Kang, L.W., Hsu, C.C., Zhuang, B., Lin, C.W., Yeh, C.H.: Learning-based joint super-resolution and deblocking for a highly compressed image. IEEE Trans. Multimed. 17(7), 921–934 (2015)
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.04948 (2018)
Li, T., He, X., Qing, L., Teng, Q., Chen, H.: An iterative framework of cascaded deblocking and super-resolution for compressed images. IEEE Trans. Multimed. 20(6), 1305–1320 (2017)
Li, Y., Guo, F., Tan, R.T., Brown, M.S.: A contrast enhancement framework with JPEG artifacts suppression. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8690, pp. 174–188. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10605-2_12
List, P., Joch, A., Lainema, J., Bjontegaard, G., Karczewicz, M.: Adaptive deblocking filter. IEEE Trans. Circ. Syst. Video Technol. 13(7), 614–619 (2003)
Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Proceedings of NIPS (2016)
Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21(12), 4695–4708 (2012)
Mittal, A., Saad, M.A., Bovik, A.C.: A completely blind video integrity oracle. IEEE Trans. Image Process. 25(1), 289–300 (2016)
Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind” image quality analyzer. IEEE Sig. Process. Lett. 20(3), 209–212 (2013)
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of CVPR, June 2018
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of ICLR (2015)
Stockhammer, T.: Dynamic adaptive streaming over HTTP-: standards and design principles. In: Proceedings of ACM MMSys, pp. 133–144. ACM (2011)
Svoboda, P., Hradis, M., Barina, D., Zemcik, P.: Compression artifacts removal using convolutional neural networks. arXiv preprint arXiv:1605.00366 (2016)
Wang, Z., Liu, D., Chang, S., Ling, Q., Yang, Y., Huang, T.S.: D3: deep dual-domain based fast restoration of JPEG-compressed images. In: Proceedings of CVPR (2016)
Wong, T.S., Bouman, C.A., Pollak, I., Fan, Z.: A document image model and estimation algorithm for optimized JPEG decompression. IEEE Trans. Image Process. 18(11), 2518–2535 (2009)
Yang, J.X., Wu, H.R.: Robust filtering technique for reduction of temporal fluctuation in H.264 video sequences. IEEE Trans. Circ. Syst. Video Technol. 20(3), 458–462 (2010)
Yang, S., Kittitornkun, S., Hu, Y.H., Nguyen, T.Q., Tull, D.L.: Blocking artifact free inverse discrete cosine transform. In: Proceedings of ICIP (2000)
Yoo, J., Lee, S.h., Kwak, N.: Image restoration by estimating frequency distribution of local patches. In: Proceedings of CVPR (2018)
Zhang, J., Xiong, R., Zhao, C., Zhang, Y., Ma, S., Gao, W.: CONCOLOR: constrained non-convex low-rank model for image deblocking. IEEE Trans. Image Process. 25(3), 1246–1259 (2016)
Zhang, X., Xiong, R., Fan, X., Ma, S., Gao, W.: Compression artifact reduction by overlapped-block transform coefficient estimation with block similarity. IEEE Trans. Image Process. 22(12), 4613–4626 (2013)
Acknowledgments
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X Pascal GPUs used for this research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Galteri, L., Seidenari, L., Bertini, M., Del Bimbo, A. (2019). Towards Real-Time Image Enhancement GANs. In: Vento, M., Percannella, G. (eds) Computer Analysis of Images and Patterns. CAIP 2019. Lecture Notes in Computer Science(), vol 11678. Springer, Cham. https://doi.org/10.1007/978-3-030-29888-3_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-29888-3_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-29887-6
Online ISBN: 978-3-030-29888-3
eBook Packages: Computer ScienceComputer Science (R0)