Skip to main content
SpringerLink
Log in

Joint regularization and low-rank fusion for atmospheric turbulence removal

  • S.I.: Human-in-the-loop Machine Learning and its Applications
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Atmospheric turbulence removal remains a challenging task, because it is very difficult to mitigate geometric distortion and remove spatially and temporally variant blur. This paper presents a novel strategy for atmospheric turbulence removal by characterizing local smoothness, nonlocal similarity and low-rank property of natural images. The main contributions are three folds. First, a joint regularization model is made which combines nonlocal total variation regularization and steering kernel regression total variation regularization in order that reference image enhancement and image registration are jointly implemented on geometric distortion reduction. Secondly, a fast split Bregman iteration algorithm is designed to address the joint variation optimization problem. Finally, a weighted nuclear norm is introduced to constrain the low-rank optimization problem to reduce blur variation and generate a fusion image. Extensive experimental results show that our method can effectively mitigate geometric deformation as well as blur variations and that it outperforms several other state-of-the-art turbulence removal methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Han M, Lyu Z, Qiu T, Xu M (2018) A review on intelligence dehazing and color restoration for underwater images. IEEE Trans Man Cybern Part A Syst Hum 50:1820–1832

    Article  Google Scholar 

  2. Yuan Q, Zhang L, Shen H (2014) Hyperspectral image denoising with a spatial spectral view fusion strategy. IEEE Trans Geosci Remote Sens 52(5):2314–2325

    Article  Google Scholar 

  3. Li W, Zhang D, Liu Z, Qiao X (2005) Fast block-based image restoration employing the improved best neighborhood matching approach. IEEE Trans Syst Man Cybern Part A Syst Hum 35(4):546–555

    Article  Google Scholar 

  4. Hong J, Cho S, Cho U (2009) A novel evolutionary approach to image enhancement filter design: method and applications. IEEE Trans Syst Man Cybern Part B Cybern 32(11):1446–1457

    Article  Google Scholar 

  5. Bue BD, Thompson DR, Eastwood M, Green RO (2015) Real-time atmoshperic correction of AVIRIS-NG imagery. IEEE Trans Geosci Remote Sens 53(12):6419–6428

    Article  Google Scholar 

  6. Zhu X, Milanfar P (2013) Removing atmospheric turbulence via space-invariant deconvolution. IEEE Trans Pattern Anal Mach Intell 35:157–170

    Article  Google Scholar 

  7. Molina R, Nunez J, Cortijo FJ, Mateos J (2001) Image restoration in astronomy: a Bayesian perspective. IEEE Signal Process Mag 18:11–29

    Article  Google Scholar 

  8. Pantin E, Starck J-L, Murtagh F (2007) Deconvolution and blind deconvolution in astronomy. In: Blind image deconvolution: theory and applications, pp 100–138

  9. Li D, Mersereau RM, Simske S (2007) Atmospheric turbulence-degraded image restoration using principal components analysis. IEEE Geosci Remote Sens Lett 4:340–344

    Article  Google Scholar 

  10. Harmeling S, Hirsch M, Sra S, Scholkopf B (2009) Online blind deconvolution for astronomical imaging. In: IEEE international conference on in computational photography, pp 1–7

  11. Tahtali M, Lambert A, Fraser D (2008) Self-tuning Kalman filter estimation of atmospheric warp. In: Proceedings of SPIE, vol 70760F

  12. Mao Y, Gilles J (2012) Non rigid geometric distortions correction-application to atmospheric turbulence stabilization. Inverse Probl Imaging 6:531–546

    Article  MathSciNet  MATH  Google Scholar 

  13. Shimizu M, Yoshimura S, Tanaka M, Okutomi M (2008) Super-resolution from image sequence under influence of hot-air optical turbulence. In: IEEE conference on computer vision and pattern recognition, pp 1–8 (2008)

  14. Zhu X, Milanfar P (2010) Image reconstruction from videos distorted by atmospheric turbulence. In: Proceedngs of SPIE, vol 7543

  15. Hirsch M, Sra S, Scholkopf B, Harmeling S (2010) Efficient filter flow for space-variant multiframe blind deconvolution. In: CVPR

  16. Roggemann MC, Stoudt CA, Welsh BM (1994) Image-spectrum signal-to-noise-ratio improvements by statistical frame selection for adaptive-optics imaging through atmospheric turbulence. Opt Eng 33:3254–3264

    Article  Google Scholar 

  17. Vorontsov MA, Carhart GW (2001) Anisoplanatic imaging through turbulent media: image recovery by local information fusion from a set of short-exposure images. J Opt Soc Am 18(6):1312–1324

    Article  Google Scholar 

  18. Tubbs RN (2003) Lucky exposures: diffraction limited astronomical imaging through the atmosphere. arXiv:astro-ph/0311481

  19. Fried DL (1978) Probability of getting a lucky short-exposure image through turbulence. J Opt Soc Am 68:1651–1657

    Article  Google Scholar 

  20. Joshi N, Cohen M, Rainier SM (2010) Lucky imaging for multi-image denoising, sharpening, and haze removal. In: IEEE international conference of computational photography

  21. Aubailly M, Vorontsov MA, Carhart GW, Valley MT (2009) Automated video enhancement from a stream of atmospherically-distorted images: the lucky-region fusion approach. In: Proceedings of SPIE—the international society for optical engineering, vol 7463

  22. Xie Y, Zhang W, Tao D, Hu W, Qu Y, Wang H (2016) Removing turbulence effect via hybrid total variation and deformation-guided kernel regression. IEEE Trans Image Process 25(10):4943–4958

    Article  MathSciNet  MATH  Google Scholar 

  23. Gu S, Zhang L, Zuo W, Feng X (2014) Weighted nuclear norm minimization with application to image denoising. In: IEEE conference on computer vision and pattern recognition, pp 2862–2869

  24. Cands EJ, Li X, Ma Y, Wright J (2011) Robust principal component analysis? J ACM 58:11

    MathSciNet  MATH  Google Scholar 

  25. Wang S, Zhang L, Liang Y (2012) Nonlocal spectral prior model for low-level vision. ACCV 2012:231–244

    Google Scholar 

  26. Mao Y, Gilles J (2012) Non rigid geometric distortions correction-application to atmospheric turbulence stabilization. Inverse Probl Imaging 6(3):531–546

    Article  MathSciNet  MATH  Google Scholar 

  27. Myronenko A, Song X (2010) Intensity-based image registration by minimizing residual complexity. IEEE Trans Med Imaging 29(11):1882–1891

    Article  Google Scholar 

  28. Takeda H, Farsiu S, Milanfar P (2007) Kernel regression for image processing and reconstruction. IEEE Trans Image Process 16:349–366

    Article  MathSciNet  Google Scholar 

  29. Goldstein T, Osher S (2009) The split Bregman method for L1-regularized problems. SIAM J Imaging Sci 2:323–343

    Article  MathSciNet  MATH  Google Scholar 

  30. Gilboa G, Osher S (2008) Nonlocal operators with applications to image processing. Multisc Model Simul 7:1005–1028

    Article  MathSciNet  MATH  Google Scholar 

  31. Xie Y, Qu Y, Tao D, Wu W, Qu Y, Zhang W (2016) Hyperspectral image restoration via sparse and low-rank matrix recovery. IEEE Trans Geosci Remote Sens 54(8):4642–4659

    Article  Google Scholar 

  32. Shan Q, Jia J, Agarwala A (2008) High-quality motion deblurring from a single image. In: ACM transactions on graphics, p 73

  33. Nagy JG, Oleary DP (1997) Fast iterative image restoration with a spatially varying PSF. In: International society for optics and photonicsin, advanced signal processing: algorithms, architectures, and implementations VII, vol 3162, pp 388–399

  34. Stockham Jr TG (1966) High-speed convolution and correlation. In: Proceedings of spring joint computer conference, pp 229–233

  35. Oreifej O, Shu G, Pace T, Shah M (2011) A two-stage reconstruction approach for seeing through water. In: IEEE conference on in computer vision and pattern recognition, pp 1153–1160

Download references

Acknowledgements

This work is supported by the National Key Research and Development Program of China No. 2020AAA0108301, the National Natural Science Foundation of China under Grant Nos. 61876161 and 61772524, the Natural Science Foundation of Shanghai (20ZR1417700), the CAAI-Huawei MindSpore Open Fund and the Fundamental Research Funds for the Central Universities.

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, School of Informatics, Xiamen University, Xiamen, China

    Yanyun Qu, Wenjin Yang & Hanzi Wang

  2. School of Computer Science and Technology, East China Normal University, Shanghai, China

    Yuan Xie

  3. Beijing Kwai Technology Co., Ltd., Beijing, China

    Weiwei Wu

  4. Applied Research Center (ARC), Tencent PCG, Shenzhen, China

    Yang Wu

Authors
  1. Yanyun Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenjin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuan Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weiwei Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hanzi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanyun Qu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qu, Y., Yang, W., Xie, Y. et al. Joint regularization and low-rank fusion for atmospheric turbulence removal. Neural Comput & Applic 35, 23369–23385 (2023). https://doi.org/10.1007/s00521-021-06336-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06336-5

Keywords

Search

Navigation