Skip to main content
SpringerLink
Log in

HRCutBlur Augment: effectively enhancing data diversity for image super-resolution

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Data augmentation is a low-cost but effective technique to suppress overfitting due to limited datasets. In this paper, we aim to design efficient data augmentation methods for image super-resolution to expand the number and diversity of data. First, we propose HRCutBlur that mixes a low-resolution image with its corresponding high-resolution image and cut-and-pastes the mixed image patch to the corresponding low-resolution image region. It alleviates the great resolution difference between low-resolution region and high-resolution region in augmented image caused by directly cut-and-pasting the high-resolution image patch to low-resolution image patch. Then, we propose HRCutBlur Augment to solve the insufficient diversity of input images caused by using a certain method alone. The core idea is to design a search space that includes various augmentation methods such as HRCutBlur and Cutout, and use a search algorithm to weighted select methods from the space to augment the input. This design can integrate the advantages of various methods and effectively enrich the diversity of data to adapt to the characteristics of different super-resolution models and various scenarios. Finally, the effectiveness and generality of our methods are verified by designing multi-dimensional experiments on different sizes of image super-resolution models and different benchmark datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

The authors confirm that most of the data supporting the findings of this study are available within the article. Other data supporting the results of this study are available from the corresponding author upon reasonable request.

References

  1. Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1122–1131 (2017) https://doi.org/10.1109/CVPRW.2017.150

  2. Ahn, N., Kang, B., Sohn, K.A.: Fast, accurate, and lightweight super-resolution with cascading residual network. In: Proceedings of the European conference on computer vision (ECCV), pp. 252–268 (2018). https://doi.org/10.1007/978-3-030-01249-6_16

  3. Antoniou, A., Storkey, A., Edwards, H. Data augmentation generative adversarial networks. arXiv preprint (2017). https://doi.org/10.48550/arXiv.1711.04340

  4. Bei, Y., Damian, A., Hu, S., et al.: New techniques for preserving global structure and denoising with low information loss in single-image super-resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 987–9877 (2018). https://doi.org/10.1109/CVPRW.2018.00132

  5. Bevilacqua, M., Roumy, A., Guillemot, C., et al.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the British Machine Vision Conference. BMVA Press, pp. 135.1–135.10 (2012). https://doi.org/10.5244/C.26.135

  6. Cai, J., Zeng, H., Yong, H., et al.: Toward real-world single image super-resolution: a new benchmark and a new model. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3086–3095, (2019). https://doi.org/10.1109/ICCV.2019.00318

  7. Chen, P., Liu, S., Zhao, H., et al.: Gridmask data augmentation. arXiv preprint (2020). https://doi.org/10.48550/arXiv.2001.04086

  8. Chen, C., Xiong, Z., Tian, X., et al.: Camera lens super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2019). https://doi.org/10.1109/CVPR.2019.00175

  9. Cubuk, E.D., Zoph, B., Shlens, J., et al.: Randaugment: practical automated data augmentation with a reduced search space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702–703 (2020). https://doi.org/10.1109/CVPRW50498.2020.00359

  10. DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv preprint (2017). https://doi.org/10.48550/arXiv.1708.04552

  11. Dong, C., Loy, C.C., He, K., et al.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision. Springer, pp. 184–199, (2014). https://doi.org/10.1109/ICCVW.2015.17

  12. Feng, R., Gu, J., Qiao, Y., et al.: Suppressing model overfitting for image super-resolution networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), (2019). https://doi.org/10.1109/cvprw.2019.00248

  13. Fujimoto, A., Ogawa, T., Yamamoto, K., et al.: Manga109 dataset and creation of metadata. In: Proceedings of the 1st International Workshop on Comics Analysis, Processing and Understanding, pp. 1–5 (2016). https://doi.org/10.1145/3011549.3011551

  14. Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al.: Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, Vol. 2. nips’14. (2014). https://doi.org/10.48550/arXiv.1406.2661

  15. Ha, V.K., Ren, J., Xu, X., et al.: Deep learning based single image super-resolution: a survey. In: International Conference on Brain Inspired Cognitive Systems, Springer, pp. 106–119, (2018). https://doi.org/10.1007/s11633-019-1183-x

  16. He, K., Sun, J.: Convolutional neural networks at constrained time cost. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5353–5360 (2015). https://doi.org/10.1109/CVPR.2015.7299173

  17. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206 (2015). https://doi.org/10.1109/CVPR.2015.7299156

  18. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016). https://doi.org/10.1109/CVPR.2016.182

  19. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015). https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  20. Lim, B., Son, S., Kim, H., et al.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 1132–1140 (2017). https://doi.org/10.1109/CVPRW.2017.151

  21. Liu, J., Liu, B., Zhou, H., et al.: Tokenmix: Rethinking image mixing for data augmentation in vision transformers. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI, Springer, pp. 455–471 (2022). https://doi.org/10.48550/arXiv.2207.08409

  22. Lugmayr, A., Danelljan, M., Timofte, R.: Unsupervised learning for real-world super-resolution. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3408–3416 (2019). https://doi.org/10.1109/ICCVW.2019.00423

  23. Mariani, G., Scheidegger, F., Istrate, R., et al.: Bagan: data augmentation with balancing gan. arXiv preprint (2018). https://doi.org/10.48550/arXiv.1803.09655

  24. Mei, Y., Fan, Y., Zhou, Y.: Image super-resolution with non-local sparse attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3517–3526 (2021). https://doi.org/10.1109/CVPR46437.2021.00352

  25. Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. J. Big Data 6(1), 1–48 (2019). https://doi.org/10.1186/s40537-019-0197-0

    Article  Google Scholar 

  26. Tong, T., Li, G., Liu, X., et al.: Image super-resolution using dense skip connections. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4809–4817 (2017). https://doi.org/10.1109/ICCV.2017.514

  27. Vu, T., Van Nguyen, C., Pham, T.X., et al.: Fast and efficient image quality enhancement via desubpixel convolutional neural networks. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2018). https://doi.org/10.1007/978-3-030-11021-5_16

  28. Wang, Z., Bovik, A.C., Sheikh, H.R., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  29. Wang, Z., Chen, J., Hoi, S.C.H.: Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(10), 3365–3387 (2021). https://doi.org/10.1109/TPAMI.2020.2982166

    Article  Google Scholar 

  30. Wu, Y., Guo, H., Chakraborty, C., et al.: Edge computing driven low-light image dynamic enhancement for object detection. IEEE Trans. Netw. Sci. Eng (2022). https://doi.org/10.1109/TNSE.2022.3151502

    Article  Google Scholar 

  31. Yoo, J., Ahn, N., Sohn, K.A.: Rethinking data augmentation for image super-resolution: a comprehensive analysis and a new strategy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8375–8384 (2020). https://doi.org/10.1109/CVPR42600.2020.00840

  32. Yun, S., Han, D., Oh, S.J., et al.: Cutmix: regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023–6032 (2019). https://doi.org/10.1109/ICCV.2019.00612

  33. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: International Conference on Curves and Surfaces, Springer, pp. 711–730 (2010). https://doi.org/10.1007/978-3-642-27413-8_47

  34. Zhang, H., Cisse, M., Dauphin, Y.N., et al.: mixup: Beyond empirical risk minimization. arXiv preprint (2017). https://doi.org/10.48550/arXiv.1710.09412

  35. Zhang, Y., Li, K., Li, K., et al.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 286–301 (2018). https://doi.org/10.1007/978-3-030-01234-2_18

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Wuhan University of Technology, Wuhan, 430070, Hubei, People’s Republic of China

    Hong Lin, Xi Wang, Chun Liu & Dewei Peng

Authors
  1. Hong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dewei Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dewei Peng.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, H., Wang, X., Liu, C. et al. HRCutBlur Augment: effectively enhancing data diversity for image super-resolution. Multimedia Systems 29, 2415–2427 (2023). https://doi.org/10.1007/s00530-023-01110-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-023-01110-0

Keywords

Search

Navigation