Skip to main content

Single Frame Atmospheric Turbulence Mitigation: A Benchmark Study and a New Physics-Inspired Transformer Model

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13679))

Included in the following conference series:

  • 3763 Accesses

Abstract

Image restoration algorithms for atmospheric turbulence are known to be much more challenging to design than traditional ones such as blur or noise because the distortion caused by the turbulence is an entanglement of spatially varying blur, geometric distortion, and sensor noise. Existing CNN-based restoration methods built upon convolutional kernels with static weights are insufficient to handle the spatially dynamical atmospheric turbulence effect. To address this problem, in this paper, we propose a physics-inspired transformer model for imaging through atmospheric turbulence. The proposed network utilizes the power of transformer blocks to jointly extract a dynamical turbulence distortion map and restore a turbulence-free image. In addition, recognizing the lack of a comprehensive dataset, we collect and present two new real-world turbulence datasets that allow for evaluation with both classical objective metrics (e.g., PSNR and SSIM) and a new task-driven metric using text recognition accuracy. The code and datasets are available at github.com/VITA-Group/TurbNet.

Z. Mao and A. Jaiswal—Equal contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Anantrasirichai, N., Achim, A., Kingsbury, N.G., Bull, D.R.: Atmospheric turbulence mitigation using complex wavelet-based fusion. IEEE Trans. Image Process. 22(6), 2398–2408 (2013). https://doi.org/10.1109/TIP.2013.2249078

  2. Chen, H., et al.: Pre-trained image processing transformer. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12294–12305 (2021)

    Google Scholar 

  3. Chimitt, N., Chan, S.H.: Simulating anisoplanatic turbulence by sampling intermodal and spatially correlated Zernike coefficients. Opt. Eng. 59(8), 1–26 (2020). https://doi.org/10.1117/1.OE.59.8.083101

  4. Cordonnier, J.B., Loukas, A., Jaggi, M.: On the relationship between self-attention and convolutional layers. arXiv preprint arXiv:1911.03584 (2019)

  5. Dosovitskiy, A., et al.: An image is worth 16 \(\times \) 16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  6. Fried, D.L.: Probability of getting a lucky short-exposure image through turbulence\(\ast \). J. Opt. Soc. Am. 68(12), 1651–1658 (1978). https://doi.org/10.1364/JOSA.68.001651, http://www.osapublishing.org/abstract.cfm?URI=josa-68-12-1651

  7. Hardie, R.C., Power, J.D., LeMaster, D.A., Droege, D.R., Gladysz, S., Bose-Pillai, S.: Simulation of anisoplanatic imaging through optical turbulence using numerical wave propagation with new validation analysis. Opt. Eng. 56(7), 1–16 (2017). https://doi.org/10.1117/1.OE.56.7.071502

  8. Hardie, R.C., Rucci, M.A., Dapore, A.J., Karch, B.K.: Block matching and wiener filtering approach to optical turbulence mitigation and its application to simulated and real imagery with quantitative error analysis. Opt. Eng. 56(7), 071503 (2017)

    Article  Google Scholar 

  9. He, R., Wang, Z., Fan, Y., Feng, D.: Atmospheric turbulence mitigation based on turbulence extraction. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1442–1446 (2016)

    Google Scholar 

  10. Hirsch, M., Sra, S., Schölkopf, B., Harmeling, S.: Efficient filter flow for space-variant multiframe blind deconvolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 607–614 (2010)

    Google Scholar 

  11. Kolmogorov, A.N.: The local structure of turbulence in incompressible viscous fluid for very large reynolds’ numbers. Akademiia Nauk SSSR Doklady 30, 301–305 (1941)

    MathSciNet  Google Scholar 

  12. Lau, C.P., Lai, Y.H., Lui, L.M.: Restoration of atmospheric turbulence-distorted images via RPCA and quasiconformal maps. Inverse Prob. (2019). https://doi.org/10.1088/1361-6420/ab0e4b. Mar

    Article  MathSciNet  MATH  Google Scholar 

  13. Lau, C.P., Lui, L.M.: Subsampled turbulence removal network. Math. Comput. Geom. Data 1(1), 1–33 (2021). https://doi.org/10.4310/MCGD.2021.v1.n1.a1

    Article  MATH  Google Scholar 

  14. Lau, C.P., Souri, H., Chellappa, R.: ATFaceGAN: Single face semantic aware image restoration and recognition from atmospheric turbulence. IEEE Trans. Biometrics Behav. Identity Sci. 1 (2021). https://doi.org/10.1109/TBIOM.2021.3058316

  15. Leonard, K.R., Howe, J., Oxford, D.E.: Simulation of atmospheric turbulence effects and mitigation algorithms on stand-off automatic facial recognition. In: Proceedings. SPIE 8546, Optics and Photonics for Counterterrorism, Crime Fighting, and Defence VIII, pp. 1–18 (2012)

    Google Scholar 

  16. Li, B., Peng, X., Wang, Z., Xu, J.Z., Feng, D.: AOD-Net: all-in-one dehazing network. In: Proceedings of the IEEE International Conference on Computer Vision (2017)

    Google Scholar 

  17. Li, B., et al.: Benchmarking single-image dehazing and beyond. IEEE Trans. Image Process. 28(1), 492–505 (2019). https://doi.org/10.1109/TIP.2018.2867951

    Article  MathSciNet  MATH  Google Scholar 

  18. Li, N., Thapa, S., Whyte, C., Reed, A.W., Jayasuriya, S., Ye, J.: Unsupervised non-rigid image distortion removal via grid deformation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2522–2532 (2021)

    Google Scholar 

  19. Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 (2021)

  20. Lou, Y., Ha Kang, S., Soatto, S., Bertozzi, A.: Video stabilization of atmospheric turbulence distortion. Inverse Probl. Imaging 7(3), 839–861 (2013). https://doi.org/10.3934/ipi.2013.7.839. Aug

    Article  MathSciNet  MATH  Google Scholar 

  21. Mao, Z., Chimitt, N., Chan, S.H.: Image reconstruction of static and dynamic scenes through anisoplanatic turbulence. IEEE Trans. Comput. Imaging 6, 1415–1428 (2020). https://doi.org/10.1109/TCI.2020.3029401

    Article  MathSciNet  Google Scholar 

  22. Mao, Z., Chimitt, N., Chan, S.H.: Accelerating atmospheric turbulence simulation via learned phase-to-space transform. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14759–14768 (2021)

    Google Scholar 

  23. Nair, N.G., Patel, V.M.: Confidence guided network for atmospheric turbulence mitigation. In: 2021 IEEE International Conference on Image Processing (ICIP), pp. 1359–1363 (2021). https://doi.org/10.1109/ICIP42928.2021.9506125

  24. Park, D., Kang, D.U., Kim, J., Chun, S.Y.: Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12351, pp. 327–343. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58539-6_20

    Chapter  Google Scholar 

  25. Roggemann, M.C., Welsh, B.M.: Imaging through Atmospheric Turbulence. Laser and Optical Science and Technology. Taylor and Francis (1996)

    Google Scholar 

  26. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  27. Schmidt, J.D.: Numerical Simulation of Optical Wave Propagation: With Examples in MATLAB. SPIE Press (2010). https://doi.org/10.1117/3.866274

  28. Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2298–2304 (2017)

    Article  Google Scholar 

  29. Tian, Z., Huang, W., He, T., He, P., Qiao, Yu.: Detecting text in natural image with connectionist text proposal network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 56–72. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_4

    Chapter  Google Scholar 

  30. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  31. Wang, X., Girshick, R.B., Gupta, A.K., He, K.: Non-local neural networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)

    Google Scholar 

  32. Wang, Z., Cun, X., Bao, J., Liu, J.: Uformer: a general u-shaped transformer for image restoration. arXiv preprint arXiv:2106.03106 (2021)

  33. Xiao, T., Dollar, P., Singh, M., Mintun, E., Darrell, T., Girshick, R.: Early convolutions help transformers see better. In: Advances in Neural Information Processing Systems, vol. 34 (2021)

    Google Scholar 

  34. Xie, Y., Zhang, W., Tao, D., Hu, W., Qu, Y., Wang, H.: Removing turbulence effect via hybrid total variation and deformation-guided kernel regression. IEEE Trans. Image Process. 25(10), 4943–4958 (2016). Oct

    Article  MathSciNet  MATH  Google Scholar 

  35. Yasarla, R., Patel, V.M.: Learning to restore a single face image degraded by atmospheric turbulence using CNNs. arXiv preprint arXiv:2007.08404 (2020)

  36. Yasarla, R., Patel, V.M.: Learning to restore images degraded by atmospheric turbulence using uncertainty. In: 2021 IEEE International Conference on Image Processing (ICIP), pp. 1694–1698 (2021). https://doi.org/10.1109/ICIP42928.2021.9506614

  37. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: efficient transformer for high-resolution image restoration. arXiv preprint arXiv:2111.09881 (2021)

  38. Zamir, S.W., et al.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14821–14831 (2021)

    Google Scholar 

  39. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. In: IEEE Trans. Pattern Anal. Mach. Intell. 40, 1452–1464 (2017)

    Google Scholar 

  40. Zhu, X., Milanfar, P.: Removing atmospheric turbulence via space-invariant deconvolution. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 157–170 (2013). https://doi.org/10.1109/TPAMI.2012.82

Download references

Acknowledgement

The research is based upon work supported in part by the Intelligence Advanced Research Projects Activity (IARPA) under Contract No. 2022–21102100004, and in part by the National Science Foundation under the grants CCSS-2030570 and IIS-2133032. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zhiyuan Mao .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mao, Z., Jaiswal, A., Wang, Z., Chan, S.H. (2022). Single Frame Atmospheric Turbulence Mitigation: A Benchmark Study and a New Physics-Inspired Transformer Model. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13679. Springer, Cham. https://doi.org/10.1007/978-3-031-19800-7_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19800-7_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19799-4

  • Online ISBN: 978-3-031-19800-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics