Skip to main content
SpringerLink
Log in

ESGAN for generating high quality enhanced samples

  • Regular Article
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Recently, convolutional neural networks (CNN) have shown significant success in image classification tasks. However, studies show that neural networks are susceptible to tiny perturbations. When the disturbing image is input, the neural network will make a different judgment. At present, most studies use the negative side of perturbation to mislead the neural network, such as adversarial examples. In this paper, considering the positive side of perturbation, we propose Enhanced Samples Generative Adversarial Networks (ESGAN) to generate high-quality enhanced samples with positive perturbation, which is designed to further improve the performance of the target classifier. Enhanced samples’ generation is composed of two parts. The super-resolution (SR) network is used to generate high visual quality images, and the noise network is used to generate positive perturbations. Our ESGAN is independent of the target classifier, so it can improve performance without retraining the classifier, thus effectively reducing the computing resources and training time of the classifier. Experiments show that the enhanced samples generated by our proposed ESGAN can effectively improve the performance of the target classifier without affecting human eye recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

The datasets generated during and analysed during the current study are available from the corresponding author on reasonable request.

Code Availability

The code of this work is available on (https://github.com/WJF20210918/ESGAN)

References

  1. Li, Q., Shen, L., Guo, S., Lai, Z.: Wavelet integrated cnns for noise-robust image classification. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7243–7252 (2020)

  2. Xie, X., Zhou, Y., Kung, S.-Y.: Exploring highly efficient compact neural networks for image classification. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 2930–2934 (2020)

  3. Sun, H., Zheng, X., Lu, X.: A supervised segmentation network for hyperspectral image classification. IEEE Trans. Image Process. 30, 2810–2825 (2021)

    Article  Google Scholar 

  4. Li, X., Wu, J., Sun, Z., Ma, Z., Cao, J., Xue, J.-H.: Bsnet: Bi-similarity network for few-shot fine-grained image classification. IEEE Trans. Image Process. 30, 1318–1331 (2021)

    Article  MathSciNet  Google Scholar 

  5. Xie, C., Wu, Y., Maaten, L.v.d., Yuille, A.L., He, K.: Feature denoising for improving adversarial robustness. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

  6. Zhang, H., Yu, Y., Jiao, J., Xing, E., Ghaoui, L.E., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: Proceedings of the 36th International Conference on Machine Learning, vol. 97, pp. 7472–7482 (2019)

  7. Jin, C., Rinard, M.: Manifold regularization for locally stable deep neural networks. arXiv preprint arXiv:2003.04286 (2020)

  8. Xie, C., Tan, M., Gong, B., Wang, J., Yuille, A.L., Le, Q.V.: Adversarial examples improve image recognition. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 816–825 (2020)

  9. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)

  10. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)

  11. Moosavi-Dezfooli, S.-M., Fawzi, A., Frossard, P.: Deepfool: a simple and accurate method to fool deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2574–2582 (2016)

  12. Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., Frossard, P.: Universal adversarial perturbations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1765–1773 (2017)

  13. Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. In: 2017 Ieee Symposium on Security and Privacy (sp), pp. 39–57 (2017). IEEE

  14. Xiao, C., Li, B., Zhu, J.-Y., He, W., Liu, M., Song, D.: Generating adversarial examples with adversarial networks. arXiv preprint arXiv:1801.02610 (2018)

  15. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

  16. Mangla, P., Jandial, S., Varshney, S., Balasubramanian, V.N.: Advgan++: Harnessing latent layers for adversary generation. arXiv preprint arXiv:1908.00706 (2019)

  17. Zhang, W., Liu, Y., Dong, C., Qiao, Y.: Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3096–3105 (2019)

  18. Wang, C., Wang, S., Xia, Z., Li, Q., Ma, B., Li, J., Yang, M., Shi, Y.-Q.: Medical image super-resolution via deep residual neural network in the shearlet domain 80, 26637–26655 (2021)

  19. Liu, J., Zhang, W., Tang, Y., Tang, J., Wu, G.: Residual feature aggregation network for image super-resolution. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2356–2365 (2020)

  20. Wang, Q., Gao, Q., Wu, L., Sun, G., Jiao, L.: Adversarial multi-path residual network for image super-resolution. IEEE Trans. Image Process. 30, 6648–6658 (2021)

    Article  Google Scholar 

  21. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision, pp. 184–199 (2014). Springer

  22. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  23. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  24. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

  25. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

  26. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Loy, C.C., Qiao, Y., Tang, X.: Esrgan: Enhanced super-resolution generative adversarial networks. European Conference on Computer Vision (2018)

  27. Rakotonirina, N.C., Rasoanaivo, A.: Esrgan+ : Further improving enhanced super-resolution generative adversarial network. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3637–3641 (2020)

  28. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  29. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  30. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  31. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009). Ieee

  32. Parkhi, O.M., Vedaldi, A., Zisserman, A., Jawahar, C.: Cats and dogs. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498–3505 (2012). IEEE

  33. Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.: Randaugment: Practical automated data augmentation with a reduced search space. Adv. Neural. Inf. Process. Syst. 33, 18613–18624 (2020)

    Google Scholar 

  34. Xie, Q., Dai, Z., Hovy, E., Luong, T., Le, Q.: Unsupervised data augmentation for consistency training. Adv. Neural. Inf. Process. Syst. 33, 6256–6268 (2020)

    Google Scholar 

  35. Berthelot, D., Carlini, N., Cubuk, E.D., Kurakin, A., Sohn, K., Zhang, H., Raffel, C.: Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In: International Conference on Learning Representations (2020)

  36. Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2017)

  37. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016)

  38. Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., Song, D.: Natural adversarial examples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15262–15271 (2021)

Download references

Funding

This work is supported by the National Natural Science Foundation of China (Grant No.62072250, 61772281, 61702235, U1636117, U1804263, 62172435, 61872203 and 61802212), the Zhongyuan Science and Technology Innovation Leading Talent Project of China (Grant No.214200510019), the Suqian Municipal Science and Technology Plan Project in 2020 (S202015), the Plan for Scientific Talent of Henan Province (Grant No.2018JR0018), the Opening Project of Guangdong Provincial Key Laboratory of Information Security Technology(Grant No.2020B1212060078), and the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD) fund.

Author information

Authors and Affiliations

  1. Nanjing University of Information Science and Technology, Nanjing, 210044, Jiangsu, China

    Junfeng Wu, Jinwei Wang & Junjie Zhao

  2. State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, 450003, Henan, China

    Jinwei Wang & Xiangyang Luo

  3. Qilu University of Technology, Jinan, 250353, Shandong, China

    Bin Ma

Authors
  1. Junfeng Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinwei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junjie Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiangyang Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bin Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by JW, JW and JZ. The first draft of the manuscript was written by XL and BM and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Jinwei Wang.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Communicated by B-K Bao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, J., Wang, J., Zhao, J. et al. ESGAN for generating high quality enhanced samples. Multimedia Systems 28, 1809–1822 (2022). https://doi.org/10.1007/s00530-022-00953-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-022-00953-3

Keywords

Search

Navigation