Abstract
Convolutional neural network (CNN) based super-resolution (SR) has achieved superior performance compared with traditional methods for uncompressed images/videos, but its performance degenerates dramatically for compressed content especially at low bit-rate scenario due to the mixture distortions during sampling and compressing. This is critical because images/videos are always compressed with degraded quality in practical scenarios. In this paper, we propose a novel dual-network structure to improve the CNN-based SR performance for compressed high definition video especially at low bit-rate. To alleviate the impact of compression, an enhancement network is proposed to remove the compression artifacts which is located ahead of the SR network. The two networks, enhancement network and SR network, are optimized stepwise for different tasks of compression artifact reduction and SR respectively. Moreover, an improved geometric self-ensemble strategy is proposed to further improve the SR performance. Extensive experimental results demonstrate that the dual-network scheme can significantly improve the quality of super-resolved images/videos compared with those reconstructed from single SR network for compressed content. It achieves around 31.5% bit-rate saving for 4 K video compression compared with HEVC when applying the proposed method in a SR-based video coding framework, which proves the potential of our method in practical scenarios, e.g., video coding and SR.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13
Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25
Jia, C., Wang, S., Zhang, X., Wang, S., Ma, S.: Spatial-temporal residue network based in-loop filter for video coding. In: 2017 IEEE Visual Communications and Image Processing (VCIP), pp. 1–4. IEEE (2017)
Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint (2016)
Liang, Y., Timofte, R., Wang, J., Gong, Y., Zheng, N.: Single image super resolution-when model adaptation matters. arXiv preprint arXiv:1703.10889 (2017)
Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, vol. 1, p. 3 (2017)
Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of the ICML, vol. 30, p. 3 (2013)
Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3791–3799 (2015)
Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)
Song, L., Tang, X., Zhang, W., Yang, X., Xia, P.: The SJTU 4K video sequence dataset. In: 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX), pp. 34–35. IEEE (2013)
Sullivan, G.J., Ohm, J., Han, W.J., Wiegand, T.: Overview of the high efficiency video coding (hevc) standard. IEEE Trans. Circuits Syst. Video Technol. 22(12), 1649–1668 (2012)
Timofte, R., et al.: Ntire 2017 challenge on single image super-resolution: methods and results. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1110–1121. IEEE (2017)
Timofte, R., De, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1920–1927. IEEE (2013)
Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 111–126. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16817-3_8
Wang, Y., Wang, L., Wang, H., Li, P.: End-to-end image super-resolution via deep and shallow convolutional networks. arXiv preprint arXiv:1607.07680 (2016)
Yang, C.Y., Yang, M.H.: Fast direct super-resolution by simple functions. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 561–568. IEEE (2013)
Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Boissonnat, J.-D., et al. (eds.) Curves and Surfaces 2010. LNCS, vol. 6920, pp. 711–730. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27413-8_47
Zhang, X., Wang, S., Zhang, Y., Lin, W., Ma, S., Gao, W.: High-efficiency image coding via near-optimal filtering. IEEE Signal Process. Lett. 24(9), 1403–1407 (2017)
Acknowledgements
This work was supported in part by National Natural Science Foundation of China (61571017), National Postdoctoral Program for Innovative Talents (BX201600006)Top-Notch Young Talents Program of China, High-performance Computing Platform of Peking University, which are gratefully acknowledged.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Feng, L., Zhang, X., Zhang, X., Wang, S., Wang, R., Ma, S. (2018). A Dual-Network Based Super-Resolution for Compressed High Definition Video. In: Hong, R., Cheng, WH., Yamasaki, T., Wang, M., Ngo, CW. (eds) Advances in Multimedia Information Processing – PCM 2018. PCM 2018. Lecture Notes in Computer Science(), vol 11164. Springer, Cham. https://doi.org/10.1007/978-3-030-00776-8_55
Download citation
DOI: https://doi.org/10.1007/978-3-030-00776-8_55
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00775-1
Online ISBN: 978-3-030-00776-8
eBook Packages: Computer ScienceComputer Science (R0)