Skip to main content
SpringerLink
Log in

EARN: toward efficient and robust JPEG compression artifact reduction

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

JPEG is one of the most widely used lossy image compression algorithms, but artifacts are generated during compression. Various artifact reduction methods have been proposed, and many of them, especially deep learning-based approaches, showed promising performance. However, one major drawback that limited their deployment and application is their cumbersome and complicated model. To remedy this problem, we propose a simple and efficient network named Efficient Artifact Reduction Network. To achieve efficiency, we consider enlarging the receptive field and preserving pixel-wise information as significant concerns. On the one hand, we notice choosing a proper down-sampling ratio is important, as the down-sampling operation is a trade-off between these two aspects. On the other hand, we design a Large Kernel Depthwise Separable Convolution block that considers both aspects. For flexibility over different compression qualities, which is the focus of research in recent years, we design a Half Adaptive Instance Normalization-based approach that elegantly integrates information of the Quantization Matrix into the feature map. It adaptively normalizes half of the channels in the Encoder to embed the compression quality information and precise pixel-wise information is preserved through the other half channels. We also design a scalable architecture inspired by prior works to enable a post-training balance between computational cost and restoration performance. Experiments on various datasets show that our network achieves state-of-the-art restoration performance with much fewer parameters and less computational cost.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data availability

The datasets used in this paper are public datasets and can be obtained by accessing websites or contacting the relevant providers.

Code Availability

The code of this study will be available at https://github.com/tgjjj/EARN.

Notes

  1. https://github.com/Lyken17/pytorch-OpCounter.

References

  1. Sheikh, H.: Live image quality assessment database release 2, http://live.ece.utexas.edu/research/quality

  2. Wallace, G.K.: The JPEG still picture compression standard. Commun. ACM 34(4), 30–44 (1991)

    Article  Google Scholar 

  3. Ehrlich, M., Davis, L., Lim, S.-N., Shrivastava, A.: Quantization guided jpeg artifact correction. In: European Conference on Computer Vision, pp. 293–309. Springer (2020)

  4. Zheng, B., Chen, Y., Tian, X., Zhou, F., Liu, X.: Implicit dual-domain convolutional network for robust color image compression artifact reduction. IEEE Trans. Circuits Syst. Video Technol. 30(11), 3982–3994 (2019)

    Article  Google Scholar 

  5. Zheng, B., Sun, R., Tian, X., Chen, Y.: S-net: a scalable convolutional neural network for JPEG compression artifact reduction. J. Electron. Imaging 27(4), 043037 (2018)

    Google Scholar 

  6. Jiang, J., Zhang, K., Timofte, R.: Towards flexible blind JPEG artifacts removal. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4997–5006 (2021)

  7. Galteri, L., Seidenari, L., Bertini, M., Del Bimbo, A.: Deep universal generative adversarial compression artifact removal. IEEE Trans. Multimedia 21(8), 2131–2145 (2019)

    Article  Google Scholar 

  8. Kim, Y., Soh, J.W., Park, J., Ahn, B., Lee, H.-S., Moon, Y.-S., Cho, N.I.: A pseudo-blind convolutional neural network for the reduction of compression artifacts. IEEE Trans. Circuits Syst. Video Technol. 30(4), 1121–1135 (2019)

    Article  Google Scholar 

  9. Kim, Y., Soh, J.W., Cho, N.I.: AGARNet: adaptively gated jpeg compression artifacts removal network for a wide range quality factor. IEEE Access 8, 20160–20170 (2020)

    Article  Google Scholar 

  10. Foi, A., Katkovnik, V., Egiazarian, K.: Pointwise shape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images. IEEE Trans. Image Process. 16(5), 1395–1411 (2007)

    Article  MathSciNet  Google Scholar 

  11. Lee, K., Kim, D.S., Kim, T.: Regression-based prediction for blocking artifact reduction in JPEG-compressed images. IEEE Trans. Image Process. 14(1), 36–48 (2004)

    MathSciNet  Google Scholar 

  12. Koç Kayhan, S.: Efficient robust filtering technique for blocking artifacts reduction. Vis. Comput. 32(4), 417–427 (2016)

    Article  Google Scholar 

  13. Golestaneh, S.A., Chandler, D.M.: Algorithm for JPEG artifact reduction via local edge regeneration. J. Electron. Imaging 23(1), 013018 (2014)

    Article  Google Scholar 

  14. Dong, C., Deng, Y., Loy, C.C., Tang, X.: Compression artifacts reduction by a deep convolutional network. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 576–584 (2015)

  15. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision, pp. 184–199. Springer (2014)

  16. Svoboda, P., Hradis, M., Barina, D., Zemcik, P.: Compression artifacts removal using convolutional neural networks. arXiv preprint arXiv:1605.00366

  17. Cavigelli, L., Hager, P., Benini, L.: CAS-CNN: a deep convolutional neural network for image compression artifact suppression. In: 2017 International Joint Conference on Neural Networks (IJCNN), IEEE, pp. 752–759 (2017)

  18. Fu, X., Zha, Z.-J., Wu, F., Ding, X., Paisley, J.: JPEG artifacts reduction via deep convolutional sparse coding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2501–2510 (2019)

  19. Chen, H., He, X., Ren, C., Qing, L., Teng, Q.: CISRDCNN: super-resolution of compressed images using deep convolutional neural networks. Neurocomputing 285, 204–219 (2018)

    Article  Google Scholar 

  20. Li, B., Shi, Y., Wang, B., Qi, Z., Liu, J.: RGSR: a two-step lossy JPG image super-resolution based on noise reduction. Neurocomputing 419, 322–334 (2021)

    Article  Google Scholar 

  21. Amaranageswarao, G., Deivalakshmi, S., Ko, S.B.: Joint restoration convolutional neural network for low-quality image super resolution. Vis. Comput. 38, 31–50 (2022). https://doi.org/10.1007/s00371-020-019

  22. Liu, P., Zhang, H., Zhang, K., Lin, L., Zuo, W.: Multi-level wavelet-CNN for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 773–782 (2018)

  23. Chen, H., He, X., Qing, L., Xiong, S., Nguyen, T.Q.: DPW-SDNet: Dual pixel-wavelet domain deep CNNs for soft decoding of JPEG-compressed images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 711–720 (2018)

  24. Florentín-Núñez, M.N., López-Rubio, E., López-Rubio, F.J.: Adaptive kernel regression and probabilistic self-organizing maps for JPEG image deblocking. Neurocomputing 121, 32–39 (2013)

    Article  Google Scholar 

  25. Liu, X., Wu, X., Zhou, J., Zhao, D.: Data-driven sparsity-based restoration of JPEG-compressed images in dual transform-pixel domain. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5171–5178 (2015)

  26. Wang, Z., Liu, D., Chang, S., Ling, Q., Yang, Y., Huang, T.S.: D3: deep dual-domain based fast restoration of jpeg-compressed images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2764–2772 (2016)

  27. Guo, J., Chao, H.: Building dual-domain representations for compression artifacts reduction. In: European Conference on Computer Vision, pp. 628–644. Springer (2016)

  28. Zhang, X., Yang, W., Hu, Y., Liu, J.: DMCNN: dual-domain multi-scale convolutional neural network for compression artifacts removal. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 390–394. IEEE (2018)

  29. Sun, M., He, X., Xiong, S., Ren, C., Li, X.: Reduction of JPEG compression artifacts based on DCT coefficients prediction. Neurocomputing 384, 335–345 (2020)

    Article  Google Scholar 

  30. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  31. Yim, C., Bovik, A.C.: Quality assessment of deblocked images. IEEE Trans. Image Process. 20(1), 88–98 (2011). https://doi.org/10.1109/TIP.2010.2061859

    Article  MathSciNet  Google Scholar 

  32. Joshi, P., Prakash, S., Rawat, S.: Continuous wavelet transform-based no-reference quality assessment of deblocked images. Vis. Comput. 34(12), 1739–1748 (2018)

    Article  Google Scholar 

  33. Li, L., Zhou, Y., Lin, W., Wu, J., Zhang, X., Chen, B.: No-reference quality assessment of deblocked images. Neurocomputing 177, 572–584 (2016)

    Article  Google Scholar 

  34. Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3883–3891 (2017)

  35. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

  36. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1501–1510 (2017)

  37. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401–4410 (2019)

  38. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Springer (2015)

  39. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  40. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122

  41. Zini, S., Bianco, S., Schettini, R.: Deep residual autoencoder for blind universal JPEG restoration. IEEE Access 8, 63283–63294 (2020). https://doi.org/10.1109/ACCESS.2020.2984387

    Article  Google Scholar 

  42. Yu, F., Wang, D., Shelhamer, E., Darrell, T.: Deep layer aggregation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2403–2412 (2018)

  43. Chen, H., He, X., Yang, H., Qing, L., Teng, Q.: A feature-enriched deep convolutional neural network for JPEG image compression artifacts reduction and its applications. IEEE Trans. Neural Netw. Learn. Syst. 33(1), 430–444 (2021)

    Article  Google Scholar 

  44. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2010)

    Article  Google Scholar 

  45. Rawzor: Image compression benchmark (2022). http://imagecompression.info/

  46. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)

  47. Maas, A.L., Hannun, A.Y., Ng, A.Y., et al.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of ICML, vol. 30, , p. 3. Citeseer (2013)

  48. Chen, H., He, X., An, C., Nguyen, T.Q.: Deep wide-activated residual network based joint blocking and color bleeding artifacts reduction for 4:2:0 JPEG-compressed images. IEEE Signal Process. Lett. 26(1), 79–83 (2019). https://doi.org/10.1109/LSP.2018.2880146

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Advanced Digital Technology and Instrument, Zhejiang University, No. 38 Zheda Road, Hangzhou, 310027, China

    Ge Teng, Rongxin Jiang, Xuesong Liu, Fan Zhou & Yaowu Chen

  2. Zhejiang Provincial Key Laboratory for Network Multimedia Technologies, Hangzhou, China

    Rongxin Jiang & Fan Zhou

  3. Zhejiang University Embedded System Engineering Research Center, Ministry of Education of China, Hangzhou, China

    Xuesong Liu & Yaowu Chen

Authors
  1. Ge Teng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rongxin Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xuesong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yaowu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rongxin Jiang.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Teng, G., Jiang, R., Liu, X. et al. EARN: toward efficient and robust JPEG compression artifact reduction. Vis Comput 40, 3033–3053 (2024). https://doi.org/10.1007/s00371-023-03008-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03008-4

Keywords

Search

Navigation