Skip to main content
SpringerLink
Log in

Weighted least square filter via deep unsupervised learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The weighted least square (WLS) filter is a popular edge-preserving image smoother that is particularly useful for detail enhancing and HDR tone mapping. However, it suffers from limited edge-preserving capability and high computational cost. Existing deep learning-based filters under the WLS framework are mostly based on supervised learning. They improve the efficiency but not the quality. In this paper, we propose a novel edge-preserving filter under the weighted least square framework based on deep unsupervised learning. According to the spatial-varying smoothing property of the edge-preserving filter, we propose a lightweight fully convolution neural network based on dilated convolutions with varying expanding factors. The proposed filter fully makes use of the 2D neighborhood information, thus it is able to suppress various artifacts. Thanks to the highly optimized framework for deep learning, the proposed filter is highly efficient, enabling the processing of 720P images at interactive rates (\(\approx \)12 fps) on a modern desktop. Experimental results indicate that the proposed filter achieves the state-of-the-art smoothing quality. Therefore, our filter benefits a variety of tasks in the field of image processing and computer graphics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

Our source code, trained model and data are available at https://github.com/dtz-dd/deepwls.

References

  1. Zhuang P, Ding X (2020) Underwater image enhancement using an edge-preserving filtering retinex algorithm. Multim Tools Appl 79(25–26):17257–17277. https://doi.org/10.1007/s11042-019-08404-4

    Article  Google Scholar 

  2. Liu K, Li X (2021) De-hazing and enhancement method for underwater and low-light images. Multim Tools Appl 80(13):19421–19439. https://doi.org/10.1007/s11042-021-10740-3

    Article  Google Scholar 

  3. Petschnigg G, Szeliski R, Agrawala M, Cohen MF, Hoppe H, Toyama K (2004) Digital photography with flash and no-flash image pairs. ACM Trans Graph 23(3):664–672. https://doi.org/10.1145/1015706.1015777

    Article  Google Scholar 

  4. Yu S, Li X, Ma M, Zhang X, Chen S (2021) Multi-focus image fusion based on L1 image transform. Multim Tools Appl 80(4):5673–5700. https://doi.org/10.1007/s11042-020-09877-4

    Article  Google Scholar 

  5. Wang X, Jin J, Yang B (2017) Diffusion map based interactive image segmentation. Multim Tools Appl 76(16):17497–17509. https://doi.org/10.1007/s11042-016-4106-7

    Article  Google Scholar 

  6. Kala R, Deepa P (2020) Adaptive fuzzy hexagonal bilateral filter for brain MRI denoising. Multim Tools Appl 79(21–22):15513–15530. https://doi.org/10.1007/s11042-019-7459-x

    Article  Google Scholar 

  7. Zhu S, Gao R, Li Z (2017) Stereo matching algorithm with guided filter and modified dynamic programming. Multim Tools Appl 76(1):199–216. https://doi.org/10.1007/s11042-015-3023-5

    Article  Google Scholar 

  8. Ding X, Chen L, Zheng X, Huang Y, Zeng D (2016) Single image rain and snow removal via guided L0 smoothing filter. Multim Tools Appl 75(5):2697–2712. https://doi.org/10.1007/s11042-015-2657-7

    Article  Google Scholar 

  9. Mondal K, Rabidas R, Dasgupta R (2022) Single image haze removal using contrast limited adaptive histogram equalization based multiscale fusion technique. Multim Tools Appl. https://doi.org/10.1007/s11042-021-11890-0

    Article  Google Scholar 

  10. Tomasi C, Manduchi R (1998) Bilateral filtering for gray and color images. In: International conference on computer vision, pp 839–846 . https://doi.org/10.1109/ICCV.1998.710815

  11. He K, Sun J, Tang X (2013) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409. https://doi.org/10.1109/TPAMI.2012.213

    Article  PubMed  Google Scholar 

  12. Qiang Z, He L, Chen Y, Chen X, Xu D (2019) Adaptive fast local laplacian filters and its edge-aware application. Multim Tools Appl 78(1):619–639. https://doi.org/10.1007/s11042-017-5347-9

    Article  Google Scholar 

  13. Liu Q, Xiong B, Yang D, Zhang M (2016) A generalized relative total variation method for image smoothing. Multim Tools Appl 75(13):7909–7930. https://doi.org/10.1007/s11042-015-2709-z

    Article  Google Scholar 

  14. Farbman Z, Fattal R, Lischinski D, Szeliski R (2008) Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans Graph 27(3):67–16710. https://doi.org/10.1145/1360612.1360666

    Article  Google Scholar 

  15. Liu W, Zhang P, Huang X, Yang J, Shen C, Reid I (2020) Real-time image smoothing via iterative least squares. ACM Trans Graph 39(3):28–12824. https://doi.org/10.1145/3388887

    Article  Google Scholar 

  16. Yang Y, Zheng H, Zeng L, Shen X, Zhan Y (2022) L1-regularized reconstruction model for edge-preserving filtering. IEEE Trans Multim, 1–1. https://doi.org/10.1109/TMM.2022.3171686

  17. Yang Y, Hui H, Zeng L, Zhao Y, Zhan Y, Yan T (2022) Edge-preserving image filtering based on soft clustering. IEEE Trans Circuits Syst Video Technol 32(7):4150–4162. https://doi.org/10.1109/TCSVT.2021.3124291

    Article  Google Scholar 

  18. Xu L, Lu C, Xu Y, Jia J (2011) Image smoothing via \(L_{0}\) gradient minimization. ACM Trans Graph 30(6):174–117411. https://doi.org/10.1145/2070781.2024208

    Article  Google Scholar 

  19. An X, Pellacini F (2008) Appprop: all-pairs appearance-space edit propagation. ACM Trans Graph 27(3):40. https://doi.org/10.1145/1360612.1360639

    Article  Google Scholar 

  20. Bhat P, Zitnick CL, Cohen MF, Curless B (2010) Gradientshop: A gradient-domain optimization framework for image and video filtering. ACM Trans Graph 29(2):10–11014. https://doi.org/10.1145/1731047.1731048

    Article  Google Scholar 

  21. Lischinski D, Farbman Z, Uyttendaele M, Szeliski R (2006) Interactive local adjustment of tonal values. ACM Trans Graph 25(3):646–653. https://doi.org/10.1145/1141911.1141936

    Article  Google Scholar 

  22. Shao P, Ding S, Ma L, Wu Y, Wu Y (2015) Edge-preserving image decomposition via joint weighted least squares. Comput Vis Media 1(1):37–47. https://doi.org/10.1007/s41095-015-0006-4

    Article  Google Scholar 

  23. Min D, Choi S, Lu J, Ham B, Sohn K, Do MN (2014) Fast global image smoothing based on weighted least squares. IEEE Trans Image Process 23(12):5638–5653. https://doi.org/10.1109/TIP.2014.2366600

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  24. Liu W, Chen X, Shen C, Liu Z, Yang J (2017) Semi-global weighted least squares in image filtering. In: International conference on computer vision, pp 5862–5870. https://doi.org/10.1109/ICCV.2017.624

  25. Chen Q, Xu J, Koltun V (2017) Fast image processing with fully-convolutional networks. In: International conference on computer vision, pp 2516–2525. https://doi.org/10.1109/ICCV.2017.273

  26. Wu H, Zheng S, Zhang J, Huang K (2018) Fast end-to-end trainable guided filter. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1838–1847. https://doi.org/10.1109/CVPR.2018.00197

  27. Xu L, Ren J, Yan Q, Liao R, Jia J (2015) Deep edge-aware filters. Proceedings of the 32nd international conference on machine learning. In: Proceedings of machine learning research, 37:1669–1678. https://proceedings.mlr.press/v37/xub15.html

  28. Yin H, Gong Y, Qiu G (2020) Fast and efficient implementation of image filtering using a side window convolutional neural network. Signal Process 176:107717. https://doi.org/10.1016/j.sigpro.2020.107717

    Article  Google Scholar 

  29. Koutis I, Miller GL, Tolliver D (2011) Combinatorial preconditioners and multilevel solvers for problems in computer vision and image processing. Comput Vis Image Underst 115(12):1638–1646. https://doi.org/10.1016/j.cviu.2011.05.013

    Article  Google Scholar 

  30. Krishnan D, Fattal R, Szeliski R (2013) Efficient preconditioning of laplacian matrices for computer graphics. ACM Trans Graph 32(4):142–114215. https://doi.org/10.1145/2461912.2461992

    Article  Google Scholar 

  31. Afonso MV, Bioucas-Dias JM, Figueiredo MAT (2010) Fast image recovery using variable splitting and constrained optimization. IEEE Trans Image Process 19(9):2345–2356. https://doi.org/10.1109/TIP.2010.2047910

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  32. Perona P, Malik J (1990) Scale-space and edge detection using anisotropic diffusion. IEEE Trans Pattern Anal Mach Intell 12(7):629–639. https://doi.org/10.1109/34.56205

    Article  Google Scholar 

  33. Barron JT, Poole B (2016) The fast bilateral solver. In: Leibe B, Matas J, Sebe N, Welling M (Eds) Computer vision - ECCV 2016. ECCV 2016. Lecture notes in computer science, vol 9907. Springer, Cham. https://doi.org/10.1007/978-3-319-46487-9_38

  34. Liu W, Zhang P, Chen X, Shen C, Huang X, Yang J (2020) Embedding bilateral filter in least squares for efficient edge-preserving image smoothing. IEEE Trans Circuits Syst Video Technol 30(1):23–35. https://doi.org/10.1109/TCSVT.2018.2890202

    Article  Google Scholar 

  35. Yu F, Koltun V (2016) Multi-scale context aggregation by dilated convolutions. In: Proceedings of the international conference on learning representations

  36. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  37. Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Proceedings of the international conference on learning representations

  38. Everingham M, Gool LV, Williams CKI, Winn JM, Zisserman A (2010) The pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303–338. https://doi.org/10.1007/s11263-009-0275-4

    Article  Google Scholar 

  39. Sun Z, Han B, Li J, Zhang J, Gao X (2020) Weighted guided image filtering with steering kernel. IEEE Trans Image Process 29:500–508. https://doi.org/10.1109/TIP.2019.2928631

    Article  ADS  MathSciNet  Google Scholar 

  40. Xu J, Liu Z, Hou Y, Zhen X, Shao L, Cheng M (2021) Pixel-level non-local image smoothing with objective evaluation. IEEE Trans Multim 23:4065–4078. https://doi.org/10.1109/TMM.2020.3037535

    Article  Google Scholar 

  41. Fan Q, Yang J, Wipf DP, Chen B, Tong X (2018) Image smoothing via unsupervised learning. ACM Trans Graph 37(6):259–125914. https://doi.org/10.1145/3272127.3275081

    Article  Google Scholar 

  42. Zhu F, Liang Z, Jia X, Zhang L, Yu Y (2019) A benchmark for edge-preserving image smoothing. IEEE Trans Image Process 28(7):3556–3570. https://doi.org/10.1109/TIP.2019.2908778

    Article  ADS  MathSciNet  Google Scholar 

  43. Feng Y, Deng S, Yan X, Yang X, Wei M, Liu L (2021) Easy2hard: Learning to solve the intractables from a synthetic dataset for structure-preserving image smoothing. IEEE Trans Neural Networks Learn Syst, 1–1. https://doi.org/10.1109/TNNLS.2021.3084473

  44. Arbelaez P, Maire M, Fowlkes CC, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell 33(5):898–916. https://doi.org/10.1109/TPAMI.2010.161

    Article  PubMed  Google Scholar 

  45. Rother C, Kolmogorov V, Blake A (2004) Grabcut: interactive foreground extraction using iterated graph cuts. ACM Trans Graph 23(3):309–314. https://doi.org/10.1145/1015706.1015720

    Article  Google Scholar 

  46. Ferrari V, Fevrier L, Jurie F, Schmid C (2008) Groups of adjacent contour segments for object detection. IEEE Trans Pattern Anal Mach Intell 30(1):36–51. https://doi.org/10.1109/TPAMI.2007.1144

    Article  CAS  PubMed  Google Scholar 

  47. Girshick RB, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587. https://doi.org/10.1109/CVPR.2014.81

  48. Canny JF (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell 8(6):679–698. https://doi.org/10.1109/TPAMI.1986.4767851

    Article  CAS  PubMed  Google Scholar 

  49. Xu L, Yan Q, Xia Y, Jia J (2012) Structure extraction from texture via relative total variation. ACM Trans Graph 31(6):139–113910. https://doi.org/10.1145/2366145.2366158

    Article  Google Scholar 

  50. Karacan L, Erdem E, Erdem A (2013) Structure-preserving image smoothing via region covariances. ACM Trans Graph 32(6):176–117611. https://doi.org/10.1145/2508363.2508403

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Jiangsu University, Xuefu, Zhenjiang, 212013, Jiangsu, China

    Yang Yang, Dan Wu & Lanling Zeng

  2. Department of Statistics and Data Science, National University of Singapore, Lower Kent Ridge, Singapore, 119077, Singapore

    Zhuoran Li

Authors
  1. Yang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lanling Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhuoran Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Yang.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, Y., Wu, D., Zeng, L. et al. Weighted least square filter via deep unsupervised learning. Multimed Tools Appl 83, 31361–31377 (2024). https://doi.org/10.1007/s11042-023-16844-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16844-2

Keywords

Search

Navigation