skip to main content
research-article

Distill-DBDGAN: Knowledge Distillation and Adversarial Learning Framework for Defocus Blur Detection

Published: 17 February 2023 Publication History

Abstract

Defocus blur detection (DBD) aims to segment the blurred regions from a given image affected by defocus blur. It is a crucial pre-processing step for various computer vision tasks. With the increasing popularity of small mobile devices, there is a need for a computationally efficient method to detect defocus blur accurately. We propose an efficient defocus blur detection method that estimates the probability of each pixel being focused or blurred in resource-constraint devices. Despite remarkable advances made by the recent deep learning-based methods, they still suffer from several challenges such as background clutter, scale sensitivity, indistinguishable low-contrast focused regions from out-of-focus blur, and especially high computational cost and memory requirement. To address the first three challenges, we develop a novel deep network that efficiently detects blur map from the input blurred image. Specifically, we integrate multi-scale features in the deep network to resolve the scale ambiguities and simultaneously modeled the non-local structural correlations in the high-level blur features. To handle the last two issues, we eventually frame our DBD algorithm to perform knowledge distillation by transferring information from the larger teacher network to a compact student network. All the networks are adversarially trained in an end-to-end manner to enforce higher order consistencies between the output and the target distributions. Experimental results demonstrate the state-of-the-art performance of the larger teacher network, while our proposed lightweight DBD model imitates the output of the teacher network without significant loss in accuracy. The codes, pre-trained model weights, and the results will be made publicly available.

References

[1]
Soonmin Bae and Frédo Durand. 2007. Defocus magnification. In Computer Graphics Forum, Vol. 26. 571–579.
[2]
Marcela Carvalho, Bertrand Le Saux, Pauline Trouvé-Peloux, Andrés Almansa, and Frédéric Champagnat. 2018. Deep depth from defocus: How can defocus blur improve 3D estimation using dense neural networks? In Proceedings of the European Conference on Computer Vision (ECCV’18) Workshops.
[3]
Ming-Ming Cheng, Niloy J. Mitra, Xiaolei Huang, Philip H. S. Torr, and Shi-Min Hu. 2014. Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37, 3 (2014), 569–582.
[4]
X. Cun and C. M. Pun. 2020. Defocus blur detection via depth distillation. In Proceedings of the European Conference Computer Vision (ECCV’20), Vol. 12358. 747–763.
[5]
J. Deng, W. Dong, R. Socher, L. J. Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’09). 248–255.
[6]
S. A. Golestaneh and L. J. Karam. 2017. Spatially-varying blur detection based on multiscale fused and sorted transform coefficients of gradient magnitudes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 596–605.
[7]
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative adversarial nets. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS’14). 2672–2680.
[8]
Wenliang Guo, Xiao Xiao, Yilong Hui, Wenming Yang, and Amir Sadovnik. 2021. Heterogeneous attention nested U-shaped network for blur detection. IEEE Signal Process. Lett. 29 (2021), 140–144.
[9]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS’17), Vol. 30.
[10]
J. Hu, L. Shen, and G. Sun. 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). 7132–7141.
[11]
R. Huang, W. Feng, M. Fan, L. Wan, and J. Sun. 2018. Multiscale blur detection by learning discriminative deep features. Neurocomputing 285 (2018), 154–166.
[12]
Zhang Jin-Yu, Chen Yan, and Huang Xian-Xiang. 2009. Edge detection of images based on improved Sobel operator and genetic algorithms. In Proceedings of the International Conference on Image Analysis and Signal Processing. 31–35.
[13]
Alexia Jolicoeur-Martineau. 2019. The relativistic discriminator: A key element missing from standard GAN. In Proceedings of the International Conference on Learning Representations (ICLR’19).
[14]
Diederick P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR’15).
[15]
Boyi Li, Felix Wu, Kilian Q. Weinberger, and Serge J. Belongie. 2019. Positional normalization. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS’19). 1620–1632.
[16]
Jinxing Li, Dandan Fan, Lingxiao Yang, Shuhang Gu, Guangming Lu, Yong Xu, and David Zhang. 2021. Layer-output guided complementary attention learning for image defocus blur detection. IEEE Trans. Image Process. 30 (2021), 3748–3763.
[17]
Zinan Lin, Vyas Sekar, and Giulia Fanti. 2020. Why spectral normalization stabilizes GANs: Analysis and improvements. Advances in Neural Information Processing Systems (NeurIPS) 34 (2020), 9652–9638.
[18]
K. Ma, H. Fu, T. Liu, Z. Wang, and D. Tao. 2018. Deep blur mapping: Exploiting high-level semantics by deep neural networks. IEEE Trans. Image Process. 27, 10 (2018), 5155–5166.
[19]
X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley. 2017. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). 2813–2821.
[20]
X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley. 2019. On the effectiveness of least squares generative adversarial networks. IEEE Trans. Pattern Anal. Mach. Intell. 41, 12 (2019), 2947–2960.
[21]
Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv:1411.1784. Retrieved from https://arxiv.org/abs/1411.1784.
[22]
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR’18).
[23]
Farzin Mokhtarian and Riku Suomela. 1998. Robust image corner detection through curvature scale space. IEEE Trans. Pattern Anal. Mach. Intell. 20, 12 (1998), 1376–1381.
[24]
Y. Pang, H. Zhu, X. Li, and X. Li. 2016. Classifying discriminative features for blur detection. IEEE Trans. Cybern. 46, 10 (2016), 2220–2227.
[25]
J. Park, Y. W. Tai, D. Cho, and I. S. Kweon. 2017. A unified approach of multi-scale deep and hand-crafted features for defocus estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 2760–2769.
[26]
Alec Radford, Luke Metz, and Soumith Chintala. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR’16), Yoshua Bengio and Yann LeCun (Eds.).
[27]
O. Ronneberger, P. Fischer, and T. Brox. 2015. U-Net: Convolutional networks for biomedical image segmentation. In Proceedings of the Annual Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI’15). 234–241.
[28]
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).
[29]
J. Shi, L. Xu, and J. Jia. 2014. Discriminative blur detection features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). 2965–2972.
[30]
J. Shi, L. Xu, and J. Jia. 2015. Just noticeable defocus blur detection and estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 657–665.
[31]
Xiaoli Sun, Xiujun Zhang, Mingqing Xiao, and Chen Xu. 2020. Blur detection via deep pyramid network with recurrent distinction enhanced modules. Neurocomputing 414 (2020), 278–290.
[32]
M. Tan and Q. V. Le. 2019. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning (ICML’19).
[33]
C. Tang, X. Liu, S. An, and P. Wang. 2021. BR\(^2\)Net: Defocus blur detection via a bidirectional channel attention residual refining network. IEEE Trans. Multimedia 23 (2021), 624–635.
[34]
C. Tang, X. Liu, X. Zheng, W. Li, J. Xiong, L. Wang, A. Zomaya, and A. Longo. 2022. DeFusionNET: Defocus blur detection via recurrently fusing and refining discriminative multi-scale deep features. IEEE Trans. Pattern Anal. Mach. Intell. 44, 2 (2022), 955–968.
[35]
Chang Tang, Xinwang Liu, Xinzhong Zhu, En Zhu, Kun Sun, Pichao Wang, Lizhe Wang, and Albert Zomaya. 2020. R\(^2\)MRF: Defocus blur detection via recurrently refining multi-scale residual features. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 12063–12070.
[36]
C. Tang, J. Wu, Y. Hou, P. Wang, and W. Li. 2016. A spectral and spatial approach of coarse-to-fine blurred image region detection. IEEE Sign. Process. Lett. 23, 11 (2016), 1652–1656.
[37]
Chang Tang, Xinzhong Zhu, Xinwang Liu, Lizhe Wang, and Albert Zomaya. 2019. DeFusionNET: Defocus blur detection via recurrently fusing and refining multi-scale deep features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19).
[38]
M. Yang, K. Yu, C. Zhang, Z. Li, and K. Yang. 2018. DenseASPP for semantic segmentation in street scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). 3684–3692.
[39]
X. Yi and M. Eramian. 2016. LBP-Based segmentation of defocus blur. IEEE Trans. Image Process. 25, 4 (2016), 1626–1638.
[40]
K. Zeng, Y. Wang, J. Mao, J. Liu, W. Peng, and N. Chen. 2019. A local metric for defocus blur detection based on CNN feature learning. IEEE Trans. Image Process. 28, 5 (2019), 2107–2115.
[41]
Yongping Zhai, Junhua Wang, Jinsheng Deng, Guanghui Yue, Wei Zhang, and Chang Tang. 2021. Global context guided hierarchically residual feature refinement network for defocus blur detection. Sign. Process. 183 (2021), 107996.
[42]
Ning Zhang and Junchi Yan. 2020. Rethinking the defocus blur detection problem and a real-time deep DBD model. In Proceedings of the European Conference Computer Vision (ECCV’20). 617–632.
[43]
Wenda Zhao, Xueqing Hou, You He, and Huchuan Lu. 2021. Defocus blur detection via boosting diversity of deep ensemble networks. IEEE Trans. Image Process. 30 (2021), 5426–5438.
[44]
Wenda Zhao, Cai Shang, and Huchuan Lu. 2021. Self-generated defocus blur detection via dual adversarial discriminators. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’21). 6933–6942.
[45]
Wenda Zhao, Fan Zhao, Dong Wang, and Huchuan Lu. 2018. Defocus blur detection via multi-stream bottom-top-bottom fully convolutional network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).
[46]
W. Zhao, F. Zhao, D. Wang, and H. Lu. 2019. Defocus blur detection via multi-stream bottom-top-bottom network. IEEE Trans. Pattern Anal. Mach. Intell. 42, 8 (2019), 1884–1897.
[47]
Wenda Zhao, Bowen Zheng, Qiuhua Lin, and Huchuan Lu. 2019. Enhancing diversity of defocus blur detectors via cross-ensemble network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19).

Cited By

View all
  • (2024)DBD-Diff: Defocus Blur Detection Using Semantic and Texture Correlation Guided Diffusion ModelProceedings of the 19th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry10.1145/3703619.3706050(1-9)Online publication date: 1-Dec-2024
  • (2024)Real-World Scene Image Enhancement with Contrastive Domain Adaptation LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/369497320:12(1-23)Online publication date: 26-Nov-2024
  • (2023)A Novel Defocus-Blur Region Detection Approach Based on DCT Feature and PCNN StructureIEEE Access10.1109/ACCESS.2023.330982011(94945-94961)Online publication date: 2023
  • Show More Cited By

Index Terms

  1. Distill-DBDGAN: Knowledge Distillation and Adversarial Learning Framework for Defocus Blur Detection

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Multimedia Computing, Communications, and Applications
    ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 2s
    April 2023
    545 pages
    ISSN:1551-6857
    EISSN:1551-6865
    DOI:10.1145/3572861
    • Editor:
    • Abdulmotaleb El Saddik
    Issue’s Table of Contents

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 17 February 2023
    Online AM: 22 August 2022
    Accepted: 03 August 2022
    Revised: 30 June 2022
    Received: 27 January 2022
    Published in TOMM Volume 19, Issue 2s

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Defocus blur detection
    2. knowledge distillation
    3. adversarial learning

    Qualifiers

    • Research-article

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)107
    • Downloads (Last 6 weeks)6
    Reflects downloads up to 20 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)DBD-Diff: Defocus Blur Detection Using Semantic and Texture Correlation Guided Diffusion ModelProceedings of the 19th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry10.1145/3703619.3706050(1-9)Online publication date: 1-Dec-2024
    • (2024)Real-World Scene Image Enhancement with Contrastive Domain Adaptation LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/369497320:12(1-23)Online publication date: 26-Nov-2024
    • (2023)A Novel Defocus-Blur Region Detection Approach Based on DCT Feature and PCNN StructureIEEE Access10.1109/ACCESS.2023.330982011(94945-94961)Online publication date: 2023
    • (2023)A Relation-Aware Network for Defocus Blur Detection2023 7th Asian Conference on Artificial Intelligence Technology (ACAIT)10.1109/ACAIT60137.2023.10528486(66-74)Online publication date: 10-Nov-2023

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    Full Text

    HTML Format

    View this article in HTML Format.

    HTML Format

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media