Abstract
Image restoration algorithms for atmospheric turbulence are known to be much more challenging to design than traditional ones such as blur or noise because the distortion caused by the turbulence is an entanglement of spatially varying blur, geometric distortion, and sensor noise. Existing CNN-based restoration methods built upon convolutional kernels with static weights are insufficient to handle the spatially dynamical atmospheric turbulence effect. To address this problem, in this paper, we propose a physics-inspired transformer model for imaging through atmospheric turbulence. The proposed network utilizes the power of transformer blocks to jointly extract a dynamical turbulence distortion map and restore a turbulence-free image. In addition, recognizing the lack of a comprehensive dataset, we collect and present two new real-world turbulence datasets that allow for evaluation with both classical objective metrics (e.g., PSNR and SSIM) and a new task-driven metric using text recognition accuracy. The code and datasets are available at github.com/VITA-Group/TurbNet.
Z. Mao and A. Jaiswal—Equal contribution.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Anantrasirichai, N., Achim, A., Kingsbury, N.G., Bull, D.R.: Atmospheric turbulence mitigation using complex wavelet-based fusion. IEEE Trans. Image Process. 22(6), 2398–2408 (2013). https://doi.org/10.1109/TIP.2013.2249078
Chen, H., et al.: Pre-trained image processing transformer. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12294–12305 (2021)
Chimitt, N., Chan, S.H.: Simulating anisoplanatic turbulence by sampling intermodal and spatially correlated Zernike coefficients. Opt. Eng. 59(8), 1–26 (2020). https://doi.org/10.1117/1.OE.59.8.083101
Cordonnier, J.B., Loukas, A., Jaggi, M.: On the relationship between self-attention and convolutional layers. arXiv preprint arXiv:1911.03584 (2019)
Dosovitskiy, A., et al.: An image is worth 16 \(\times \) 16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Fried, D.L.: Probability of getting a lucky short-exposure image through turbulence\(\ast \). J. Opt. Soc. Am. 68(12), 1651–1658 (1978). https://doi.org/10.1364/JOSA.68.001651, http://www.osapublishing.org/abstract.cfm?URI=josa-68-12-1651
Hardie, R.C., Power, J.D., LeMaster, D.A., Droege, D.R., Gladysz, S., Bose-Pillai, S.: Simulation of anisoplanatic imaging through optical turbulence using numerical wave propagation with new validation analysis. Opt. Eng. 56(7), 1–16 (2017). https://doi.org/10.1117/1.OE.56.7.071502
Hardie, R.C., Rucci, M.A., Dapore, A.J., Karch, B.K.: Block matching and wiener filtering approach to optical turbulence mitigation and its application to simulated and real imagery with quantitative error analysis. Opt. Eng. 56(7), 071503 (2017)
He, R., Wang, Z., Fan, Y., Feng, D.: Atmospheric turbulence mitigation based on turbulence extraction. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1442–1446 (2016)
Hirsch, M., Sra, S., Schölkopf, B., Harmeling, S.: Efficient filter flow for space-variant multiframe blind deconvolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 607–614 (2010)
Kolmogorov, A.N.: The local structure of turbulence in incompressible viscous fluid for very large reynolds’ numbers. Akademiia Nauk SSSR Doklady 30, 301–305 (1941)
Lau, C.P., Lai, Y.H., Lui, L.M.: Restoration of atmospheric turbulence-distorted images via RPCA and quasiconformal maps. Inverse Prob. (2019). https://doi.org/10.1088/1361-6420/ab0e4b. Mar
Lau, C.P., Lui, L.M.: Subsampled turbulence removal network. Math. Comput. Geom. Data 1(1), 1–33 (2021). https://doi.org/10.4310/MCGD.2021.v1.n1.a1
Lau, C.P., Souri, H., Chellappa, R.: ATFaceGAN: Single face semantic aware image restoration and recognition from atmospheric turbulence. IEEE Trans. Biometrics Behav. Identity Sci. 1 (2021). https://doi.org/10.1109/TBIOM.2021.3058316
Leonard, K.R., Howe, J., Oxford, D.E.: Simulation of atmospheric turbulence effects and mitigation algorithms on stand-off automatic facial recognition. In: Proceedings. SPIE 8546, Optics and Photonics for Counterterrorism, Crime Fighting, and Defence VIII, pp. 1–18 (2012)
Li, B., Peng, X., Wang, Z., Xu, J.Z., Feng, D.: AOD-Net: all-in-one dehazing network. In: Proceedings of the IEEE International Conference on Computer Vision (2017)
Li, B., et al.: Benchmarking single-image dehazing and beyond. IEEE Trans. Image Process. 28(1), 492–505 (2019). https://doi.org/10.1109/TIP.2018.2867951
Li, N., Thapa, S., Whyte, C., Reed, A.W., Jayasuriya, S., Ye, J.: Unsupervised non-rigid image distortion removal via grid deformation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2522–2532 (2021)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 (2021)
Lou, Y., Ha Kang, S., Soatto, S., Bertozzi, A.: Video stabilization of atmospheric turbulence distortion. Inverse Probl. Imaging 7(3), 839–861 (2013). https://doi.org/10.3934/ipi.2013.7.839. Aug
Mao, Z., Chimitt, N., Chan, S.H.: Image reconstruction of static and dynamic scenes through anisoplanatic turbulence. IEEE Trans. Comput. Imaging 6, 1415–1428 (2020). https://doi.org/10.1109/TCI.2020.3029401
Mao, Z., Chimitt, N., Chan, S.H.: Accelerating atmospheric turbulence simulation via learned phase-to-space transform. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14759–14768 (2021)
Nair, N.G., Patel, V.M.: Confidence guided network for atmospheric turbulence mitigation. In: 2021 IEEE International Conference on Image Processing (ICIP), pp. 1359–1363 (2021). https://doi.org/10.1109/ICIP42928.2021.9506125
Park, D., Kang, D.U., Kim, J., Chun, S.Y.: Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12351, pp. 327–343. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58539-6_20
Roggemann, M.C., Welsh, B.M.: Imaging through Atmospheric Turbulence. Laser and Optical Science and Technology. Taylor and Francis (1996)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Schmidt, J.D.: Numerical Simulation of Optical Wave Propagation: With Examples in MATLAB. SPIE Press (2010). https://doi.org/10.1117/3.866274
Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2298–2304 (2017)
Tian, Z., Huang, W., He, T., He, P., Qiao, Yu.: Detecting text in natural image with connectionist text proposal network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 56–72. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_4
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Wang, X., Girshick, R.B., Gupta, A.K., He, K.: Non-local neural networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)
Wang, Z., Cun, X., Bao, J., Liu, J.: Uformer: a general u-shaped transformer for image restoration. arXiv preprint arXiv:2106.03106 (2021)
Xiao, T., Dollar, P., Singh, M., Mintun, E., Darrell, T., Girshick, R.: Early convolutions help transformers see better. In: Advances in Neural Information Processing Systems, vol. 34 (2021)
Xie, Y., Zhang, W., Tao, D., Hu, W., Qu, Y., Wang, H.: Removing turbulence effect via hybrid total variation and deformation-guided kernel regression. IEEE Trans. Image Process. 25(10), 4943–4958 (2016). Oct
Yasarla, R., Patel, V.M.: Learning to restore a single face image degraded by atmospheric turbulence using CNNs. arXiv preprint arXiv:2007.08404 (2020)
Yasarla, R., Patel, V.M.: Learning to restore images degraded by atmospheric turbulence using uncertainty. In: 2021 IEEE International Conference on Image Processing (ICIP), pp. 1694–1698 (2021). https://doi.org/10.1109/ICIP42928.2021.9506614
Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: efficient transformer for high-resolution image restoration. arXiv preprint arXiv:2111.09881 (2021)
Zamir, S.W., et al.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14821–14831 (2021)
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. In: IEEE Trans. Pattern Anal. Mach. Intell. 40, 1452–1464 (2017)
Zhu, X., Milanfar, P.: Removing atmospheric turbulence via space-invariant deconvolution. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 157–170 (2013). https://doi.org/10.1109/TPAMI.2012.82
Acknowledgement
The research is based upon work supported in part by the Intelligence Advanced Research Projects Activity (IARPA) under Contract No. 2022–21102100004, and in part by the National Science Foundation under the grants CCSS-2030570 and IIS-2133032. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Mao, Z., Jaiswal, A., Wang, Z., Chan, S.H. (2022). Single Frame Atmospheric Turbulence Mitigation: A Benchmark Study and a New Physics-Inspired Transformer Model. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13679. Springer, Cham. https://doi.org/10.1007/978-3-031-19800-7_25
Download citation
DOI: https://doi.org/10.1007/978-3-031-19800-7_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-19799-4
Online ISBN: 978-3-031-19800-7
eBook Packages: Computer ScienceComputer Science (R0)