Skip to main content
SpringerLink
Log in

A super-resolution enhancement of UAV images based on a convolutional neural network for mobile devices

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

The task of reconstructing high-resolution images given low-resolution images is called image super-resolution (SR). Within the wide variety of RS applications, it has a strong impact on the advancement of unmanned aerial vehicle (UAV) technologies. The camera is one of the most commonly used sensors in current UAVs, allowing complex information to be collected from the environments to be monitored. However, UAVs usually have limitations regarding their flight time. If one wants to monitor a large geographic area in a short time, flights must be performed at higher altitudes. This implies a loss of spatial resolution because cameras have limitations in terms of their optics and the size of the pixels. In recent years, SR techniques based on convolutional neural networks (CNNs) that are able to learn the correspondence between low-resolution images and their high-resolution counterparts have been developed. Taking advantage of advances in smartphones in terms of popularity and computing capabilities, the aim of this paper is to demonstrate an architecture for the SR of images captured by a UAV. The images are sent from a low-resolution camera in a UAV to a mobile device from which it is possible to obtain the corresponding SR image. A benchmark dataset of images is used for both quantitative and qualitative assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Baker S, Kanade T (2000) Hallucinating faces. In: Proceedings Fourth IEEE international conference on automatic face and gesture recognition (Cat. No. PR00580), pp 83–88. https://doi.org/10.1109/AFGR.2000.840616

  2. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. IEEE, pp 248–255

  3. Dengwen Z (2010) An edge-directed bicubic interpolation algorithm. In: 2010 3rd international congress on image and signal processing, vol 3, pp 1186–1189. https://doi.org/10.1109/CISP.2010.5647190

  4. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2):295–307. https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  5. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2):295–307. https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  6. Farrell J (2008) Aided navigation: GPS with high rate sensors. McGraw-Hill Inc, New York

    Google Scholar 

  7. Fasano G, Accado D, Moccia A, Moroney D (2016) Sense and avoid for unmanned aircraft systems. IEEE Aerosp. Electron. Syst. Mag. 31(11):82–110

    Article  Google Scholar 

  8. Ferrick A, Fish J, Venator E, Lee GS (2012) Uav obstacle avoidance using image processing techniques. In: 2012 IEEE international conference on technologies for practical robot applications (TePRA). IEEE, pp 73–78

  9. Freeman WT, Pasztor EC, Carmichael OT (2000) Learning low-level vision. Intl J Comput Vis

  10. Gao JJ, Chen XH, Li JY, Liu GC, Ma J (2010) Irregular seismic data reconstruction based on exponential threshold model of pocs method. Appl. Geophys. 7(3):229–238

    Article  Google Scholar 

  11. Garcia J, Molina JM (2018) Analysis of sensor fusion solutions for uav. In: Conferencia de la asociación Española para la inteligencia artificial (CAEPIA), pp 23–26

  12. Gibbins D, Roberts P, Swierkowski L (2004) A video geo-location and image enhancement tool for small unmanned air vehicles (uavs). In: Proceedings of the 2004 intelligent sensors, sensor networks and information processing conference, 2004. IEEE, pp 469–473

  13. Greenspan H (2008) Super-resolution in medical imaging. Comput. J. 52(1):43–63

    Article  Google Scholar 

  14. Haris M, Watanabe T, Fan L, Widyanto MR, Nobuhara H (2017) Superresolution for uav images via adaptive multiple sparse representation and its application to 3-d reconstruction. IEEE Trans. Geosci. Remote Sens. 55(7):4047–4058

    Article  Google Scholar 

  15. Instruments N (2019) Peak signal-to-noise ratio as an image quality metric. http://www.ni.com/es-es/innovations/white-papers/11/peak-signal-to-noise-ratio-as-an-image-quality-metric.html

  16. Johnson DH (2006) Signal-to-noise ratio. Scholarpedia 1(12):2088

    Article  Google Scholar 

  17. Kim J, Kwon Lee J, Mu Lee K (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654

  18. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  19. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690

  20. Lei J, Zhang S, Luo L, Xiao J, Wang H (2018) Super-resolution enhancement of uav images based on fractional calculus and pocs. Geo-spatial Information Science 21(1):56–66

    Article  Google Scholar 

  21. Liu C, Sun D (2014) On Bayesian adaptive video super resolution. IEEE Trans Pattern Anal Mach Intell 36(2):346–360

    Article  Google Scholar 

  22. Mareboyana M, Le Moigne J (2018) Super-resolution of remote sensing images using edge-directed radial basis functions. In: Signal processing, sensor/information fusion, and target recognition XXVII, vol 10646, p 1064610. International society for optics and photonics

  23. Mjolsness E (1985) Fingerprint hallucination. Diss California Institute of Technology

  24. Ng MK, Shen H, Lam EY, Zhang L (2007) A total variation regularization based super-resolution reconstruction algorithm for digital video. EURASIP J Adv Signal Process 2007(1):074585

    Article  Google Scholar 

  25. Nguyen N, Milanfar P, Golub G (2001) A computationally efficient superresolution image reconstruction algorithm. IEEE Trans Image Process 10(4):573–583

    Article  Google Scholar 

  26. Noda H, Niimi M (2007) Colorization in ycbcr color space and its application to jpeg images. Pattern Recogn 40(12):3714–3720. https://doi.org/10.1016/j.patcog.2007.04.005

    Article  MATH  Google Scholar 

  27. Rankin G, Tirkel A, Leukhin A (2015) Millimeter wave array for uav imaging. In: MIMO radar radar symposium (IRS)

  28. Seibel H, Goldenstein S, Rocha A (2017) Eyes on the target: Super-resolution and license-plate recognition in low-quality surveillance videos. IEEE Access 5:20020–20035

    Article  Google Scholar 

  29. Sheikh HR, Bovik AC, De Veciana G (2005) An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans Image Process 14(12):2117–2128

    Article  Google Scholar 

  30. Siyah B (2019) Semantic drone dataset. https://www.kaggle.com/bulentsiyah/semantic-drone-dataset

  31. Timofte R, De V, Gool LV (2013) Anchored neighborhood regression for fast example-based super-resolution. In: 2013 IEEE international conference on computer vision, pp 1920–1927. https://doi.org/10.1109/ICCV.2013.241

  32. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  33. Xiao J, Liu T, Zhang Y, Zou B, Lei J, Li Q (2016) Multi-focus image fusion based on depth extraction with inhomogeneous diffusion equation. Signal Processing 125:171–186. https://doi.org/10.1016/j.sigpro.2016.01.014. http://www.sciencedirect.com/science/article/pii/S0165168416000311

    Article  Google Scholar 

  34. Yamanaka J, Kuwashima S, Kurita T (2017) Fast and accurate image super resolution by deep cnn with skip connection and network in network. In: Liu D, Xie S, Li Y, Zhao D, El-Alfy ESM (eds) Neural Information Processing. Springer International Publishing, Cham, pp 217–225

Download references

Funding

This work was funded by the private research project of Company BQ and the public research projects of the Spanish Ministry of Economy and Competitiveness (MINECO), references TEC2017-88048-C2-2-R, RTC-2016-5595-2, RTC-2016-5191-8, and RTC-2016-5059-8.

Author information

Authors and Affiliations

  1. BQ Engineering Team, Calle Sofía, 10. Európolis, Las Rozas, 28232, Madrid, Spain

    Daniel González

  2. Applied Artificial Intelligence Group (GIAA), Universidad Carlos III de Madrid, Madrid, Spain

    Miguel A. Patricio, Antonio Berlanga & José M. Molina

Authors
  1. Daniel González
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Miguel A. Patricio
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Antonio Berlanga
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. José M. Molina
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miguel A. Patricio.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

González, D., Patricio, M.A., Berlanga, A. et al. A super-resolution enhancement of UAV images based on a convolutional neural network for mobile devices. Pers Ubiquit Comput 26, 1193–1204 (2022). https://doi.org/10.1007/s00779-019-01355-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-019-01355-5

Keywords

Search

Navigation