Skip to main content

Restore Anything with Masks: Leveraging Mask Image Modeling for Blind All-in-One Image Restoration

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15103))

Included in the following conference series:

  • 236 Accesses

Abstract

All-in-one image restoration aims to handle multiple degradation types using one model. This paper proposes a simple pipeline for all-in-one blind image restoration to Restore Anything with Masks (RAM). We focus on the image content by utilizing Mask Image Modeling to extract intrinsic image information rather than distinguishing degradation types like other methods. Our pipeline consists of two stages: masked image pre-training and fine-tuning with mask attribute conductance. We design a straightforward masking pre-training approach specifically tailored for all-in-one image restoration. This approach enhances networks to prioritize the extraction of image content priors from various degradations, resulting in a more balanced performance across different restoration tasks and achieving stronger overall results. To bridge the gap of input integrity while preserving learned image priors as much as possible, we selectively fine-tuned a small portion of the layers. Specifically, the importance of each layer is ranked by the proposed Mask Attribute Conductance (MAC), and the layers with higher contributions are selected for finetuning. Extensive experiments demonstrate that our method achieves state-of-the-art performance. Our code and model will be released at https://github.com/Dragonisss/RAM.

Chu-Jie Qin: A part of this work is done during Chu-Jie Qin’s internship at Samsung.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Chen, H., et al.: Pre-trained image processing transformer. In: CVPR, pp. 12299–12310 (2021)

    Google Scholar 

  2. Chen, H., et al.: Masked image training for generalizable deep image denoising. In: CVPR, pp. 1692–1703 (2023)

    Google Scholar 

  3. Chen, L., Chu, X., Zhang, X., Sun, J.: Simple baselines for image restoration. In: ECCV, pp. 17–33. Springer (2022)

    Google Scholar 

  4. Chen, W.T., Huang, Z.K., Tsai, C.C., Yang, H.H., Ding, J.J., Kuo, S.Y.: Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: toward a unified model. In: CVPR, pp. 17653–17662 (2022)

    Google Scholar 

  5. Wei, C., Wang, W., Yang, W., Liu, J.: Deep retinex decomposition for low-light enhancement. In: BMVC. British Machine VLOLision Association (2018)

    Google Scholar 

  6. Dhamdhere, K., Sundararajan, M., Yan, Q.: How important is a neuron. In: ICLR (2019)

    Google Scholar 

  7. Duan, H., et al.: Masked autoencoders as image processors. arXiv preprint arXiv:2303.17316 (2023)

  8. Fan, Q., Chen, D., Yuan, L., Hua, G., Yu, N., Chen, B.: A general decoupled learning framework for parameterized image operators. PAMI 43(1), 33–47 (2019)

    Article  Google Scholar 

  9. Fang, Y., Zhang, H., Wong, H.S., Zeng, T.: A robust non-blind deblurring method using deep denoiser prior. In: CVPRW, pp. 735–744 (June 2022)

    Google Scholar 

  10. Fu, X., Huang, J., Ding, X., Liao, Y., Paisley, J.: Clearing the skies: a deep network architecture for single-image rain removal. TIP 26(6), 2944–2956 (2017)

    MathSciNet  Google Scholar 

  11. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: CVPR, pp. 3855–3863 (2017)

    Google Scholar 

  12. Gu, J., Dong, C.: Interpreting super-resolution networks with local attribution maps. In: CVPR, pp. 9199–9208 (2021)

    Google Scholar 

  13. Guo, C.L., Yan, Q., Anwar, S., Cong, R., Ren, W., Li, C.: Image dehazing transformer with transmission-aware 3d position embedding. In: CVPR, pp. 5812–5820 (2022)

    Google Scholar 

  14. Guo, C., et al.: Zero-reference deep curve estimation for low-light image enhancement. In: CVPR, pp. 1780–1789 (2020)

    Google Scholar 

  15. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick., R.: Masked autoencoders are scalable vision learners. In: CVPR, pp. 16000–16009 (2022)

    Google Scholar 

  16. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: CVPR, pp. 5197–5206 (2015)

    Google Scholar 

  17. Jin, X., Han, L.H., Li, Z., Guo, C.L., Chai, Z., Li, C.: Dnf: Decouple and feedback network for seeing in the dark. In: CVPR, pp. 18135–18144 (2023)

    Google Scholar 

  18. Kenton, J.D.M.W.C., Toutanova, L.K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: NAACL, pp. 4171–4186 (2019)

    Google Scholar 

  19. Leino, K., Sen, S., Datta, A., Fredrikson, M., Li, L.: Influence-directed explanations for deep convolutional networks. In: ITC, pp. 1–8. IEEE (2018)

    Google Scholar 

  20. Li, B., et al.: Benchmarking single-image dehazing and beyond. TIP 28(1), 492–505 (2018)

    Google Scholar 

  21. Li, B., Liu, X., Hu, P., Wu, Z., Lv, J., Peng, X.: All-in-one image restoration for unknown corruption. In: CVPR, pp. 17452–17462 (2022)

    Google Scholar 

  22. Li, C., et al.: Embedding fourier for ultra-high-definition low-light image enhancement. In: ICLR (2022)

    Google Scholar 

  23. Li, D., Zhang, Y., Cheung, K.C., Wang, X., Qin, H., Li, H.: Learning degradation representations for image deblurring. In: ECCV, pp. 736–753. Springer (2022)

    Google Scholar 

  24. Li, R., Tan, R.T., Cheong, L.F.: All in one bad weather removal using architectural search. In: CVPR, pp. 3175–3185 (2020)

    Google Scholar 

  25. Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: ECCV, pp. 254–269 (2018)

    Google Scholar 

  26. Li, Y., et al.: Lsdir: a large scale dataset for image restoration. In: CVPR, pp. 1775–1787 (2023)

    Google Scholar 

  27. Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: CVPR (2016)

    Google Scholar 

  28. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: image restoration using swin transformer. In: CVPR, pp. 1833–1844 (2021)

    Google Scholar 

  29. Lin, X., Ren, C., Liu, X., Huang, J., Lei, Y.: Unsupervised image denoising in real-world scenarios via self-collaboration parallel generative adversarial branches. In: ICCV, pp. 12642–12652 (2023)

    Google Scholar 

  30. Lin, X., Yue, J., Ren, C., Guo, C.L., Li, C.: Unlocking low-light-rainy image restoration by pairwise degradation feature vector guidance. arXiv preprint arXiv:2305.03997 (2023)

  31. Liu, L., et al.: Tape: Task-agnostic prior embedding for image restoration. In: ECCV, pp. 447–464. Springer (2022)

    Google Scholar 

  32. Liu, Y., He, J., Gu, J., Kong, X., Qiao, Y., Dong, C.: Degae: a new pretraining paradigm for low-level vision. In: CVPR, pp. 23292–23303 (2023)

    Google Scholar 

  33. Liu, Y., et al.: Discovering distinctive “semantics” in super-resolution networks. arXiv preprint arXiv:2108.00406 (2021)

  34. Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: CVPR, pp. 10012–10022 (2021)

    Google Scholar 

  35. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: ICCV, pp. 3397–3405 (2015)

    Google Scholar 

  36. Luo, Y., et al.: Mowe: mixture of weather experts for multiple adverse weather removal. arXiv preprint arXiv:2303.13739 (2023)

  37. Magid, S.A., Lin, Z., Wei, D., Zhang, Y., Gu, J., Pfister, H.: Texture-based error analysis for image super-resolution. In: CVPR, pp. 2118–2127 (2022)

    Google Scholar 

  38. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, vol. 2, pp. 416–423. IEEE (2001)

    Google Scholar 

  39. Mehri, A., Ardakani, P.B., Sappa, A.D.: Mprnet: multi-path residual network for lightweight image super resolution. In: CVPR, pp. 2704–2713 (2021)

    Google Scholar 

  40. Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: CVPR (2017)

    Google Scholar 

  41. Park, D., Lee, B.H., Chun, S.Y.: All-in-one image restoration for unknown degradations using adaptive discriminative filters for specific degradations. In: CVPR, pp. 5815–5824 (2023)

    Google Scholar 

  42. Potlapalli, V., Zamir, S.W., Khan, S.H., Shahbaz Khan, F.: Promptir: prompting for all-in-one image restoration. NeurIPS 36 (2024)

    Google Scholar 

  43. Radford, A., et al.: Improving language understanding by generative pre-training (2018)

    Google Scholar 

  44. Shrikumar, A., Su, J., Kundaje, A.: Computationally efficient measures of internal neuron importance. arXiv preprint arXiv:1807.09946 (2018)

  45. Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: ICML, pp. 3319–3328. PMLR (2017)

    Google Scholar 

  46. Sundararajan, M., Taly, A., Yan, Q.: Gradients of counterfactuals. ICLR (2017)

    Google Scholar 

  47. Wang, X., Wang, W., Cao, Y., Shen, C., Huang, T.: Images speak in images: a generalist painter for in-context visual learning. In: CVPR, pp. 6830–6839 (2023)

    Google Scholar 

  48. Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., Li, H.: Uformer: a general u-shaped transformer for image restoration. In: CVPR, pp. 17683–17693 (2022)

    Google Scholar 

  49. Wu, R.Q., Duan, Z.P., Guo, C.L., Chai, Z., Li, C.: Ridcp: revitalizing real image dehazing via high-quality codebook priors. In: CVPR, pp. 22282–22291 (2023)

    Google Scholar 

  50. Xie, L., Wang, X., Dong, C., Qi, Z., Shan, Y.: Finding discriminative filters for specific degradations in blind super-resolution. NeurIPS 34, 51–61 (2021)

    Google Scholar 

  51. Xie, Z., et al.: Simmim: a simple framework for masked image modeling. In: CVPR, pp. 9653–9663 (2022)

    Google Scholar 

  52. Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: CVPR, pp. 1357–1366 (2017)

    Google Scholar 

  53. Yang, W., Tan, R.T., Wang, S., Fang, Y., Liu, J.: Single image deraining: from model-based to data-driven and beyond. PAMI 43(11), 4059–4077 (2020)

    Article  Google Scholar 

  54. Yang, W., Wang, W., Huang, H., Wang, S., Liu, J.: Sparse gradient regularized deep retinex network for robust low-light image enhancement. TIP 30, 2072–2086 (2021)

    Google Scholar 

  55. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: Efficient transformer for high-resolution image restoration. In: CVPR, pp. 5728–5739 (2022)

    Google Scholar 

  56. Zhang, C., Zhu, Y., Yan, Q., Sun, J., Zhang, Y.: All-in-one multi-degradation image restoration network via hierarchical degradation representation. In: ACMMM, pp. 2285–2293 (2023)

    Google Scholar 

  57. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: CVPR, pp. 695–704 (2018)

    Google Scholar 

  58. Zhang, H., Sindagi, V., Patel, V.M.: Image de-raining using a conditional generative adversarial network. TCSVT 30(11), 3943–3956 (2019)

    Google Scholar 

  59. Zhang, J., et al.: Ingredient-oriented multi-degradation learning for image restoration. In: CVPR, pp. 5825–5835 (2023)

    Google Scholar 

  60. Zheng, N., et al.: Empowering low-light image enhancer through customized learnable priors. In: ICCV, pp. 12559–12569 (2023)

    Google Scholar 

  61. Zhu, Y., et al.: Learning weather-general and weather-specific features for image restoration under multiple adverse weather conditions. In: CVPR, pp. 21747–21758 (2023)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Chongyi Li .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 11714 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Qin, CJ. et al. (2025). Restore Anything with Masks: Leveraging Mask Image Modeling for Blind All-in-One Image Restoration. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15103. Springer, Cham. https://doi.org/10.1007/978-3-031-72995-9_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72995-9_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72994-2

  • Online ISBN: 978-3-031-72995-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics