Skip to main content
SpringerLink
Log in

Learning adaptive hyper-guidance via proxy-based bilevel optimization for image enhancement

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In recent years, image enhancement based on deep network plays a vital role and has become the mainstream research. However, current approaches are generally limited to the manual embedding of auxiliary components (e.g., hyper-parameters, appended modules) to train the network; thus, they can often lack flexibility, adaptability, or even fail to achieve the optimal settings. Moreover, the straightforward learning-based architectures cannot adequately handle the complex latent image distributions in real-world scenarios. To partially address the above issues, in this work, a generic adaptive hyper-training scheme based on bilevel optimization is established. Specifically, we propose a completely new bilevel deep-unfolded strategy to collaboratively optimize the inner-level task-related hyper-guidance and the outer-level image reconstruction. The process can embed the differentiable proxy-based network with parameters to automatically learn the appended control mechanism. Instead of constructing the empirically manual interventions, our strategy can proactively learn to learn self-adaptive auxiliary modules. Extensive experiments demonstrate the superiority of our strategy to address different image enhancement tasks (i.e., image restoration, image rain removal and image haze removal) in terms of flexibility and effectiveness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. It is known that the simple bi-level optimization is just a specific bilevel optimization problem with only one variable [35] ?,.

References

  1. Simoes, M., Almeida, L.B., Bioucas-Dias, J., Chanussot, J.: A framework for fast image deconvolution with incomplete observations. IEEE Trans. Image Process. 25(11), 5266–5280 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  2. Wang, Y., Yang, J., Yin, W., Zhang, Y.: A new alternating minimization algorithm for total variation image reconstruction. SIAM J. Imaging Sci. 1(3), 248–272 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  3. Cheng, J., Gao, Y., Guo, B., Zuo, W.: Image restoration using spatially variant hyper-laplacian prior. Signal Image Video Process. 13(1), 155–162 (2019)

    Article  Google Scholar 

  4. Liu, D., Wen, B., Fan, Y., Loy, C. C., Huang, T. S.: Non-local recurrent network for image restoration. In NeurIPS (2018)

  5. Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep cnn denoiser prior for image restoration. In CVPR, (2017)

  6. Tirer, T., Giryes, R.: Image restoration by iterative denoising and backward projections. IEEE Trans. Image Process. 28(3), 1220–1234 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  7. Kruse, J., Rother, C., Schmidt, U.: Learning to push the limits of efficient fft-based image deconvolution. In ICCV, (2017)

  8. Zhang, K., Zuo, W., Zhang, L.: Ffdnet: toward a fast and flexible solution for cnn-based image denoising. IEEE Trans. Image Process. 27(9), 4608–4622 (2018)

    Article  MathSciNet  Google Scholar 

  9. Liu, R., Jiang, Z., Fan, X., Luo, Z.: Knowledge-driven deep unrolling for robust image layer separation. IEEE Trans. Neural Netw. Learn. Syst. 31(5), 1653–1666 (2019)

    Article  Google Scholar 

  10. Zhang, K., Gool, L. V., Timofte, R.: Deep unfolding network for image super-resolution. In CVPR (2020)

  11. Liu, R., Cheng, S., Ma, L., Fan, X., Luo, Z.: Deep proximal unrolling: algorithmic framework, convergence analysis and applications. IEEE Trans. Image Process. 28(10), 5013–5026 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  12. Liu, R., Lin, Z., Zhang, W., Su, Z.: Learning pdes for image restoration via optimal control. In ECCV, (2010)

  13. Tai, X. C., Lie, K. A., Chan, T. F., Osher, S.: Image processing based on partial differential equations. In Proceedings of the International Conference on PDE-Based Image Processing and Related Inverse Problems. Springer Science and Business Media, Berlin (2006)

  14. Liu, R., Cheng, S., He, Y., Fan, X., Lin, Z., Luo, Z.: On the convergence of learning-based iterative methods for nonconvex inverse problems. IEEE Trans. Pattern Anal. Mach. Intell. 42(12), 3027–3039 (2019)

    Article  Google Scholar 

  15. Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In CVPR, (2018)

  16. Ren, W., Zhang, J., Ma, L., Pan, J., Cao, X., Zuo, W., Liu, W., Yang, M.H.: Deep non-blind econvolution via generalized low-rank approximation. In NeurIPS (2018)

  17. Cai, J., Zuo, W., Zhang, L.: Extreme channel prior embedded network for dynamic scene deblurring. arXiv:1903.00763 (2019)

  18. Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M. H., Shao, L.: Cycleisp: Real image restoration via improved data synthesis. arXiv:2003.07761, (2020)

  19. Guo, Y., Chen, J., Wang, J., Chen, Q., Cao, J., Deng, Z., Xu, Y., Tan, M.: Closed-loop matters: Dual regression networks for single image super-resolution. arXiv:2003.07018 (2020)

  20. Liu, R., Fan, X., Hou, M., Jiang, Z., Luo, Z., Zhang, L.: Learning aggregated transmission propagation networks for haze removal and beyond. IEEE Trans. Neural Netw. Learn. Syst. 30(10), 2973–2986 (2018)

    Article  Google Scholar 

  21. Liu, R., Ma, L., Wang, Y., Zhang, L.: Learning converged propagations with deep prior ensemble for image enhancement. IEEE Trans. Image Process. 28(3), 1528–1543 (2018)

    Article  MathSciNet  Google Scholar 

  22. Liu, R., Cheng, S., Liu, X., Ma, L., Fan, X., Luo, Z.: A bridging framework for model optimization and deep propagation. In NeurIPS (2018)

  23. Li, Y., Tan, R. T., Guo, X., Lu, J., Brown, M. S.: Rain streak removal using layer priors. In CVPR (2016)

  24. Xiaojie Guo, Yu., Li, J.M., Ling, H.: Mutually guided image filtering. IEEE Trans. Neural Netw. Learn. Syst. 42(3), 694–707 (2020)

    Google Scholar 

  25. Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In CVPR (2019)

  26. Zhang, H., Patel, V. M.: Density-aware single image de-raining using a multi-stream dense network. In CVPR (2018)

  27. Du, Y., Xu, J., Qiu, Q., Zhen, X., Zhang, L.: Variational image deraining. In WACV (2020)

  28. Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: a better and simpler baseline. In CVPR (2019)

  29. Liu, X., Ma, Y., Shi, Z., Chen, J.: Griddehazenet: attention-based multi-scale network for image dehazing. In ICCV (2019)

  30. MacKay, M., Vicol, P., Lorraine, J., Duvenaud, D., Grosse, R.: Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. arXiv:1903.03088 (2019)

  31. Dempe, S.: Foundations of Bilevel Programming. Springer Science and Business Media, Berlin (2002)

    MATH  Google Scholar 

  32. Dempe, S., Dutta, J., Mordukhovich, B.S.: New necessary optimality conditions in optimistic bilevel programming. Optimization 56(5–6), 577–604 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  33. Kohli, B.: Optimality conditions for optimistic bilevel programming problem using convexifactors. J. Optim. Theory Appl. 152(3), 632–651 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  34. Lampariello, L., Sagratella, S.: Numerically tractable optimistic bilevel problems. Comput. Optim. Appl. 76(2), 277–303 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  35. Liu, R., Mu, P., Yuan, X., Zeng, S., Zhang, J.: A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton. In ICML (2020)

  36. Liu, R., Mu, P., Yuan, X., Zeng, S., Zhang, J.: A generic descent aggregation framework for gradient-based bi-level optimization. In ICML (2021)

  37. Liu, R., Liu, X., Yuan, X., Zeng, S., Zhang, J.: A value-function-based interior-point method for non-convex bi-level optimization (2021)

  38. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. JMLR 13(2), 281–305 (2012)

    MathSciNet  MATH  Google Scholar 

  39. Swersky, K., Snoek, J., Adams, R. P.: Freeze-thaw bayesian optimization. arXiv:1406.3896 (2014)

  40. Bottou, L.: Large-scale machine learning with stochastic gradient descent. In COMPSTAT (2010)

  41. Kingma, D. P., Ba, J.: A method for stochastic optimization. arXiv:1412.6980, (2014)

  42. Lorraine, J., Duvenaud, D.: Stochastic hyperparameter optimization through hypernetworks arXiv:1802.09419 (2018)

  43. Franceschi, L., Frasconi, P., Salzo, S., Grazzi, R., Pontil, M.: Bilevel programming for hyperparameter optimization and meta-learning. In ICML (2018)

  44. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In CVPR (2017)

  45. Fan, Z., Wu, H., Fu, X., Hunag, Y., Ding, X.: Residual-guide feature fusion network for single image deraining. arXiv:1804.07493 (2018)

  46. Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In CVPR (2017)

  47. Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In ECCV (2018)

  48. Narasimhan, S.G., Nayar, S.K: Chromatic framework for vision in bad weather. In CVPR (2000)

  49. He, K., Sun, J., Tang, X.: Single image haze removal using dark channel prior. IEEE Trans. Neural Netw. Learn. Syst. 33(12), 2341–2353 (2010)

    Google Scholar 

  50. Cai, B., Xiangmin, X., Jia, K., Qing, C., Tao, D.: Dehazenet: an end-to-end system for single image haze removal. IEEE Trans. Image Process. 25(11), 5187–5198 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  51. Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., Yang, M. H.: Single image dehazing via multi-scale convolutional neural networks. In ECCV (2016)

  52. Li, B., Peng, X., Wang, Z., Xu, J., Feng, D.: Aod-net: All-in-one dehazing network. In ICCV (2017)

  53. Qu, Y., Chen, Y., Huang, J., Xie, Y.: Enhanced pix2pix dehazing network. In CVPR, (2019)

  54. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2010)

    Article  Google Scholar 

  55. Schmidt, U., Jancsary, J., Nowozin, S., Roth, S., Rother, C.: Cascades of regression tree fields for image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 38(4), 677–689 (2015)

    Article  Google Scholar 

  56. Levin, A., Weiss, Y., Durand, F., Freeman, W.T.: Understanding and evaluating blind deconvolution algorithms. In CVPR (2009)

  57. Sun, L., Cho, S., Wang, J., Hays, J.: Edge-based blur kernel estimation using patch priors. In ICCP (2013)

  58. Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-laplacian priors. In NeurIPS (2009)

  59. Schuler, C. J., Christopher Burger, H., Harmeling, S., Scholkopf, B.: A machine learning approach for non-blind image deconvolution. In CVPR (2013)

  60. Pan, J., Lin, Z., Su, Z., Yang, M. H. Robust kernel estimation with outliers handling for image deblurring. In CVPR (2016)

  61. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  62. Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., Lau, R. W.: Spatial attentive single-image deraining with a high quality real rain dataset. In CVPR (2019)

  63. Li, B., Ren, W., Dengpan, F., Tao, D., Feng, D., Zeng, W., Wang, Z.: Benchmarking single-image dehazing and beyond. IEEE Trans. Image Process. 28(1), 492–505 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  64. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In ECCV, (2012)

  65. Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In CVPR (2003)

  66. Liu, F., Shen, C., Lin, G., Reid, I.: Learning depth from single monocular images using deep convolutional neural fields. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 2024–2039 (2015)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Software Technology, Dalian University of Technology, Dalian, China

    Jiaxin Gao & Xiaokun Liu

  2. International School of Information Science and Engineering, Dalian University of Technology, Dalian, China

    Risheng Liu & Xin Fan

  3. Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province, Dalian, China

    Jiaxin Gao, Xiaokun Liu, Risheng Liu & Xin Fan

Authors
  1. Jiaxin Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaokun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Risheng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xin Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Risheng Liu.

Ethics declarations

Conflict of interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, and there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled, “Learning Adaptive Hyper-guidance via Proxy-based Bilevel Optimization for Image Restoration”.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, J., Liu, X., Liu, R. et al. Learning adaptive hyper-guidance via proxy-based bilevel optimization for image enhancement. Vis Comput 39, 1471–1484 (2023). https://doi.org/10.1007/s00371-022-02423-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02423-3

Keywords

Search

Navigation