Abstract
As an important display mode of underwater environments, the sonar image has limitations on the resolution, which often leads to problems with low resolution of underwater objects. Therefore, the image super-resolution algorithm is needed to transform the images from low-resolution to high-resolution. It can improve the visual effect and contribute to subsequent processing such as 3D reconstruction and object recognition. This paper proposes a method for sonar image super-resolution based on generative adversarial networks (GAN). By comparing the super-resolution effects of various interpolation and convolutional neural network algorithms on sonar images, a Residual-in-Residual Dense Block network is employed as the generator of GAN since it has the low distortion and high perceptual quality. Because the sonar image training set does not have enough data, the generator utilizes the transfer learning on the sonar images to produce an optimized network model which is more suitable for super-resolution of sonar image. The VGG19 network is employed as the discriminator. In addition, the perceptual loss is introduced into the loss function of \(\hbox {S}^2\hbox {RGAN}\) to further improve the perceptual quality of super-resolution images. The experimental results indicate that the proposed \(\hbox {S}^2\hbox {RGAN}\) shows excellent performance. The generated super-resolution images of \(\hbox {S}^2\hbox {RGAN}\) have the remarkable advantages of both lower distortion and higher perceptual quality comparing with other methods. Because \(\hbox {S}^2\hbox {RGAN}\) focuses more on the reality and overall visual effect of super-resolution sonar images, it is suitable for various underwater situations.
Similar content being viewed by others
References
Zhang, X., Tang, M., Tong, R.: Robust super resolution of compressed video. Vis. Comput. 28(12), 1167–1180 (2012)
Pandey, G., Ghanekar, U.: Classification of priors and regularization techniques appurtenant to single image super-resolution. Vis. Comput. 36(6), 1291–1304 (2019). https://doi.org/10.1007/s00371-019-01729-z
XuXin, K., Yang, W.X., He, S., Zhang, Q., Yin, B., Wei, X., Lau, R.W.H.: Efficient image super-resolution integration. Vis. Comput. 34(6–8), 1065–1076 (2018)
Mikaeli, E., Aghagolzadeh, A., Azghani, M.: Single-image super-resolution via patch-based and group-based local smoothness modeling. Vis. Comput. 1–17 (2019). https://doi.org/10.1007/s00371-019-01756-w
Keys, R.: Cubic convolution interpolation for digital image processing. IEEE Trans. Acoust. Speech Signal Process. 29(6), 1153–1160 (1981)
Stark, H., Oskoui, P.: High resolution image recovery from low resolution detector arrays. In: Digital Image Synthesis and Inverse Optics, pp. 80–84 (2010)
Yu, J., Xiao, C., Su, K.: A method of gibbs artifact reduction for POCS super-resolution image reconstruction. In: International Conference on Signal Processing, pp. 1–4 (2006)
Irani, M., Peleg, S.: Improving resolution by image registration. CVGIP Graph. Models Image Process. 53(3), 231–239 (1991)
Dai, S., Han, M., Wu, Y., et al.: Bilateral back-projection for single image super resolution. In: IEEE International Conference on Multimedia and Expo, pp. 1039–1042 (2007)
Rasti, P., Demirel, H., Anbarjafari, G.: Improved iterative back projection for video super-resolution. In: Signal Processing and Communications Applications Conference, pp. 552–555 (2014)
Park, J., Ku, B., Jin, Y., et al.: Side scan sonar image super resolution via region-selective sparse coding. IEICE Trans. Inf. Syst. 102(1), 210–213 (2019)
Hurtós, N., Petillot, Y., Salvi, J.: Fourier-based registrations for two-dimensional forward-looking sonar image mosaicing. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5298–5305 (2012)
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., et al.: Generative adversarial nets. In: Annual Conference on Neural Information Processing Systems, pp. 2672–2680 (2014)
Dong, C., Loy, C.C., He, K., et al.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)
Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision, pp. 391–407 (2016)
Shi, W., Caballero, J., Husz, F., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)
Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)
He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Lim, B., Son, S., Kim, H., et al.: Enhanced deep residual networks for single image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1132–1140 (2017)
Wang, Y., Perazzi, F., McWilliams, B., Sorkine-Hornung, A., Sorkine-Hornung, O., Schroers, C.: A fully progressive approach to single-image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 977–986 (2018)
Mirza, M., Osindero, S.: Conditional generative adversarial nets. In: Computer Science, pp. 2672–2680 (2014)
Isola, P., Zhu, J.Y., Zhou, T., Alexei, A., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5967–5976 (2017)
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 105–114 (2017)
Wang, X., Ke, Y., Wu, S., Gu, J., Liu, Y., Dong, C., Change Loy, C., Qiao, Y., Tang, X.: ESRGAN: enhanced super-resolution generative adversarial networks. In: European Conference on Computer Vision, pp. 63–79 (2019)
Sung, M., Joe, H., Kim, J., Yu, S.-C.: Convolutional neural network based resolution enhancement of underwater sonar image without losing working range of sonar sensors. IN: OCEANS-MTS/IEEE Kobe Techno-Oceans, pp. 1–6 (2018)
Sung, M., Kim, J., Yu, S.C.: Image-based super resolution of underwater sonar images using generative adversarial network. In: TENCON 2018 IEEE Region 10 Conference, pp. 0457-0461 (2018)
Blau, Y., Michaeli, T.: The perception-distortion tradeoff. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6228–6237 (2018)
Blau, Y., Mechrez, R., Timofte, R., et al.: The 2018 PIRM challenge on perceptual image super-resolution. In: European Conference on Computer Vision, pp. 334–355 (2019)
Huang, G., Liu, Z., Van Der Maaten, L., et al.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Shi, W., Caballero, J., Huszár, F., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. IN: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)
Timofte, R., Agustsson, E., Van Gool, L., et al.: Ntire 2017 challenge on single image super-resolution: methods and results. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 114–125 (2017)
Deng, J., Dong, W., Socher, R., et al.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv preprint arXiv:1409.1556
Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind” image quality analyzer. IEEE Signal Process. Lett. 20(3), 209–212 (2013)
Ma, C., Yang, C.-Y., Yang, X., et al.: Learning a no-reference quality metric for single-image super-resolution. Comput. Vis. Image Underst. 158, 1–16 (2017)
Kim, J., Kwon Lee, J., Mu, Lee K.: Deeply-recursive convolutional network for image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)
Isola, P., Zhu, J.Y., Zhou, T., et al.: Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
https://pan.baidu.com/s/10BOafbgHVXrbVqtVUP3Dvw Password: 6xs5 (2020)
Acknowledgements
This work is supported by National Natural Science Foundation of China (No. 61501132), China Postdoctoral Science Foundation (No. 2019M661319), Heilong-jiang Postdoctoral Scientific Research Developmental Founda-tion (No. LBH-Q17042), and Fundamental Research Funds for the Central Universities (No. 3072019CFT0603).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Song, H., Wang, M., Zhang, L. et al. \(\hbox {S}^2\hbox {RGAN}\): sonar-image super-resolution based on generative adversarial network. Vis Comput 37, 2285–2299 (2021). https://doi.org/10.1007/s00371-020-01986-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-020-01986-3