Skip to main content
SpringerLink
Log in

NasmamSR: a fast image super-resolution network based on neural architecture search and multiple attention mechanism

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Although the current super-resolution model based on deep learning has achieved excellent reconstruction results, the increasing depth of the model results in huge parameters, limiting the further application of the super-resolution deep model. To solve this problem, we propose an efficient super-resolution model based on neural architecture search and attention mechanism. First, we use global residual learning to limit the search to the non-linear mapping part of the network and add a down-sampling to this part to reduce the feature map’s size and computation. Second, we establish a lightweight search space and joint rewards for searching the optimal network structure. The model divides the search into macro search and micro search, which are used to search for the optimal down-sampling position and the optimal cell structure, respectively. In addition, we introduce the Bayesian algorithm for hyper-parameter tuning and further improve the model’s performance based on the optimal sub-network searched out. Detailed experiments show that our model achieves excellent super-resolution performance and high computational efficiency compared with some state-of-the-art models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Yan, C., Gong, B., Wei, Y., et al.: Deep multi-view enhancement hashing for image retrieval[J] IEEE Trans. Pattern Anal Mach Intell 43(4), 1445–1451 (2020)

    Article  Google Scholar 

  2. Yan, C., Li, Z., Zhang, Y., et al.: Depth image denoising using nuclear norm and learning graph model[J]. ACM Trans Mult Comput Commun Appl (TOMM) 16(4), 1–17 (2020)

    Article  Google Scholar 

  3. Yan C, Hao Y, Li L, et al. (2021) Task-adaptive attention for image captioning[J]. IEEE Trans Circuits Syst Video Technol.

  4. Vijayvergia, A., Kumar, K.: Selective shallow models strength integration for emotion detection using GloVe and LSTM[J]. Multimed Tools Appl 1, 1–15 (2021)

    Google Scholar 

  5. Yan C, Teng T, Liu Y, et al. (2021) Precise no-reference image quality evaluation based on distortion identification[J]. ACM Trans Multimed Comput Commun Appl (TOMM).

  6. Kumar K, Shrimankar DD (2018) ESUMM: event summarization on scale-free networks[J]. IETE Tech Rev.

  7. Kumar, K.: EVS-DK: event video skimming using deep keyframe[J]. J Vis Commun Image Rep 58, 345–352 (2019)

    Article  Google Scholar 

  8. Yang X, Li Z, Guo Y, et al. (2021) Retinal vessel segmentation based on an improved deep forest[J]. Internat J Imaging Syst Technol.

  9. Sharma, S., Kumar, K.: ASL-3DCNN: American sign language recognition technique using 3-D convolutional neural networks[J]. Multimed Tools Appl 1, 1–13 (2021)

    Google Scholar 

  10. Sun J, Xu Z, Shum HY (2008) Image super-resolution using gradient profile prior[C]//2008 IEEE conference on computer vision and pattern recognition. IEEE 1–8.

  11. Yan, Q., Xu, Y., Yang, X., et al.: Single image super resolution based on gradient profile sharpness[J]. IEEE Trans Image Proc 24(10), 3187–3202 (2015)

    Article  Google Scholar 

  12. Yang X, Liu L, Zhu C, et al. (2020) An improved anchor neighborhood regression SR method based on low-rank constraint[J]. The Visual Comput 1–14.

  13. Zhang K, Gool LV, Timofte R (2020) Deep unfolding network for image super-resolution[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3217–3226.

  14. Ji X, Cao Y, Tai Y, et al. (2020) Real-world super-resolution via kernel estimation and noise injection[C]//proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 466–467.

  15. Kumar, K., Shrimankar, D.D.: F-DES: fast and deep event summarization[J]. IEEE Trans Multimed 20(2), 323–334 (2017)

    Article  Google Scholar 

  16. Yan C, Meng L, Li L, et al. (2021) Age-invariant face recognition by multi-feature fusion and decomposition with self-attention[J]. ACM Trans Multimed Comput Commun Appl (TOMM).

  17. Kumar, K., Shrimankar, D.D.: Deep event learning boost-up approach: delta[J]. Multimed Tools Appl 77(20), 26635–26655 (2018)

    Article  Google Scholar 

  18. Kumar, K., Shrimankar, D.D., Singh, N.: Eratosthenes sieve based key-frame extraction technique for event summarization in videos[J]. Multimed Tools Appl 77(6), 7383–7404 (2018)

    Article  Google Scholar 

  19. Dong C, Loy CC, He K, et al. (2014) Learning a deep convolutional network for image super-resolution[C]//European conference on computer vision. Springer, Cham

  20. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 1646–1654.

  21. Shi W, Caballero J, Huszár F, et al. (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1874–1883.

  22. Ledig C, Theis L, Huszár F, et al. (2017) Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4681–4690.

  23. Lim B, Son S, Kim H, et al. (2017) Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 136–144.

  24. Yang X, Zhang Y, Li T, et al. (2021) Image super-resolution based on the down-sampling iterative module and deep CNN[J]. Circuits Syst Signal Proc. pp. 1–19.

  25. Shi W, Du H, Mei W, et al. (2020) (SARN) spatial-wise attention residual network for image super-resolution[J]. Visual Comput pp. 1–12.

  26. Yang, X., Li, X., Li, Z., et al.: Image super-resolution based on deep neural network of multiple attention mechanism[J]. J Visual Commun Image Rep 75, 103019 (2021)

    Article  Google Scholar 

  27. Tian, C., Zhuge, R., Wu, Z., et al.: Lightweight image super-resolution with enhanced CNN[J]. Knowledge-Based Syst 205, 106235 (2020)

    Article  Google Scholar 

  28. Wei P, Xie Z, Lu H, et al. (2020) Component divide-and-conquer for real-world image super-resolution[C]//European conference on computer vision. Springer, Cham pp. 101–117.

  29. Kumar, K.: Text query based summarized event searching interface system using deep learning over cloud[J]. Multimed Tools Appl 80(7), 11079–11094 (2021)

    Article  Google Scholar 

  30. Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning[J]. arXiv arXiv, 1611.01578 (2016)

    Google Scholar 

  31. Zoph B, Vasudevan V, Shlens J, et al. (2017) Learning transferable architectures for scalable image recognition[J].

  32. Pham H, Guan MY, Zoph B, et al. (2018) Efficient neural architecture search via parameter sharing[J].

  33. Weng, Y., Chen, Z., Zhou, T.: Improved differentiable neural architecture search for single image super-resolution[J]. Peer-to-Peer Netw Appl 14(3), 1806–1815 (2021)

    Article  Google Scholar 

  34. Chu X, Zhang B, Ma H, et al. (2019) Fast, accurate and lightweight super-resolution with neural architecture search[J]. arXiv: 190107261. (arXiv preprint)

  35. Krishna R, Kumar K (2020) P-MEC: polynomial congruence based multimedia encryption technique over cloud[J]. IEEE Consumer Electronics Magazine.

  36. Guo Y, Luo Y, He Z, et al. (2020) Hierarchical neural architecture search for single image super-resolution[J]. arXiv: 200304619. (arXiv preprint)

  37. Ahn N, Kang B, Sohn KA (2018) Fast, accurate, and lightweight super-resolution with cascading residual network[C]//Proceedings of the European conference on computer vision (ECCV). 252–268.

  38. Bevilacqua M, Roumy A, Guillemot C, et al. (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding[J]. 135–131.

  39. Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations[C]//International conference on curves and surfaces. Springer, Berlin, Heidelberg. pp. 711–730.

  40. Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics, in ICCV.

  41. Huang JB, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5197–5206.

  42. Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network[C]//European conference on computer vision. Springer, Cham, 391–407.

  43. Jiang K, Wang Z, Yi P, et al. (2020) Hierarchical dense recursive network for image super-resolution[J]. Pat Recognit 107:107475.

  44. Kim J, Kwon LJ, Mu LK (2016) Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1637–1645.

  45. Hui Z, Wang X, Gao X (2018) Fast and accurate single image super-resolution via information distillation network[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 723–731.

  46. Lai W S, Huang J B, Ahuja N, et al. (2017) Deep laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 624–632.

  47. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3147–3155.

  48. Zhang K, Zuo W, Zhang L (2018) Learning a single convolutional super-resolution network for multiple degradations[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3262–3271.

Download references

Acknowledgements

This research was supported by the National Natural Science Foundation of China (61573182, 62073164), and by the Fundamental Research Funds for the Central Universities (NS2020025).

Funding

The National Natural Science Foundation of China (61573182, 62073164), and by the Fundamental Research Funds for the Central Universities (NS2020025).

Author information

Authors and Affiliations

  1. College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, China

    Xin Yang, Jiangfeng Fan, Chenhuan Wu, Dake Zhou & Tao Li

Authors
  1. Xin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiangfeng Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chenhuan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dake Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Yang.

Additional information

Communicated by C. Yan.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, X., Fan, J., Wu, C. et al. NasmamSR: a fast image super-resolution network based on neural architecture search and multiple attention mechanism. Multimedia Systems 28, 321–334 (2022). https://doi.org/10.1007/s00530-021-00841-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-021-00841-2

Keywords

Search

Navigation