Skip to main content
SpringerLink
Log in

Color–depth multi-task learning for object detection in haze

  • S.I. : Multi-Source Data Understanding (MSDU)
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Haze environments pose serious challenges for object detection, making existing methods difficult to generate satisfied results. However, there is no escape from haze environments in real-world applications, especially in water and bad weather. Hence, it is necessary to enable object detection methods to conquer the difficulties caused by the haze effect. In spite of the diversity between various conditions, haze environments share a common characteristic that the haze concentration is changed with the scene depth. Hence, this haze concentration feature can be used as a representation of the scene depth. This provides us a novel cue available for object detection in haze that the object-background depth contrast can be identified. In this paper, we propose a multi-task learning-based object detection method by jointly using the color and depth features. A pair of background models is built separately with the color and depth features, forming two streams of our multi-task learning framework. The final object detection results are generated by fusing the results given by color and depth features. In contrast to existing object detection methods, the novelty of our method lies in the combination of the color and depth features under a unified multi-task learning mechanism, which is experimentally demonstrated to be robust against challenging haze environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Gazzah S, Mhalla A, Amara NEB (2017) Vehicle detection on a video traffic scene: Review and new perspectives. In: 2017 International conference on information and digital technologies (IDT). IEEE, pp 448–454

  2. Tu Z, Zheng A, Yang E, Luo B, Hussain A (2015) A biologically inspired vision-based approach for detecting multiple moving objects in complex outdoor scenes. Cogn Comput 7(5):539–551

    Article  Google Scholar 

  3. Yang Z, Yu W, Liang P, Guo H, Xia L, Zhang F, Ma J (2018) Deep transfer learning for military object recognition under small training set condition. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3468-3

    Article  Google Scholar 

  4. Angelopoulou A, Rodriguez JG, Orts-Escolano S, Gupta G, Psarrou A (2018) Fast 2D/3D object representation with growing neural gas. Neural Comput Appl 29(10):903–919

    Article  Google Scholar 

  5. Wang Y, Jodoin PM, Porikli F, Konrad J, Benezeth Y, Ishwar P (2014) CDnet 2014: an expanded change detection benchmark dataset. In: , 2014 IEEE conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 393–400

  6. Zhu Y, Chang L, Dai J, Zheng H, Zheng B (2016) Automatic object detection and segmentation from underwater images via saliency-based region merging. In OCEANS 2016-Shanghai. IEEE, pp 1–4

  7. Wang N, Zheng B, Zheng H, Yu Z (2017) Feeble object detection of underwater images through LSR with delay loop. Opt Express 25(19):22490–22498

    Article  Google Scholar 

  8. Li X, Yang Z, Shang M, Hao J (2016) Underwater image enhancement via dark channel prior and luminance adjustment. In: OCEANS 2016-Shanghai. IEEE, pp 1–5

  9. Wang JB, He N, Zhang LL, Lu K (2015) Single image dehazing with a physical model and dark channel prior. Neurocomputing 149:718–728

    Article  Google Scholar 

  10. Lee S, Yun S, Nam JH, Won CS, Jung SW (2016) A review on dark channel prior based image dehazing algorithms. EURASIP J Image Video Process 2016(1):1–4

    Article  Google Scholar 

  11. Li CY, Guo JC, Cong RM, Pang YW, Wang B (2016) Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior. IEEE Trans Image Process 25(12):5664–5677

    Article  MathSciNet  MATH  Google Scholar 

  12. Nayar SK, Narasimhan SG (1999) Vision in bad weather. In: The proceedings of the seventh IEEE international conference on computer vision, vol 2. IEEE, pp 820–827

  13. Narasimhan SG, Nayar SK (2003) Contrast restoration of weather degraded images. IEEE Trans Pattern Anal Mach Intell 25(6):713–724

    Article  Google Scholar 

  14. Song H et al (2017) Modeling of a dynamic dual-input dual-output fast steeringmirror system. Front Inf Technol Electronic Eng 18(10):1488–1498

    Article  Google Scholar 

  15. Tan RT (2008) Visibility in bad weather from a single image. In: IEEE conference on computer vision and pattern recognition. CVPR 2008. IEEE, pp 1–8

  16. Fattal R (2008) Single image dehazing. ACM Trans Graph (TOG) 27(3):72–78

    Article  Google Scholar 

  17. Chen JKBNB, Cohen-Or MCD, Deussen O, Lischinski MUD (2008) Deep photo: model-based photograph enhancement and viewing. ACM Trans Graph 27(5):32–39

    Article  Google Scholar 

  18. Tarel JP, Hautiere N (2009) Fast visibility restoration from a single color or gray level image. In: 2009 IEEE 12th international conference on computer vision. IEEE, pp 2201–2208

  19. Nishino K, Kratz L, Lombardi S (2012) Bayesian defogging. Int J Comput Vis 98(3):263–278

    Article  MathSciNet  Google Scholar 

  20. Meng G, Wang Y, Duan J, Xiang S, Pan C (2013) Efficient image dehazing with boundary constraint and contextual regularization. In: 2013 IEEE international conference on computer vision (ICCV). IEEE, pp 617–624

  21. He K, Sun J, Tang X (2011) Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell 33(12):2341–2353

    Article  Google Scholar 

  22. Cai B, Xu X, Tao D (2016) Real-time video dehazing based on spatio-temporal mrf. In: Pacific rim conference on multimedia. Springer, Cham, pp 315–325

    Google Scholar 

  23. Wang G, Ren G, Jiang L, Quan T (2013) Single image dehazing algorithm based on sky region segmentation. Inf Technol J 12(6):1168–1175

    Article  Google Scholar 

  24. Yu F, Qing C, Xu X, Cai B (2016) Image and video dehazing using view-based cluster segmentation. In: Visual communications and image processing (VCIP). IEEE, pp 1–4

  25. Yoon I, Jeong S, Jeong J, Seo D, Paik J (2015) Wavelength-adaptive dehazing using histogram merging-based classification for UAV images. Sensors 15(3):6633–6651

    Article  Google Scholar 

  26. Zhu Y, Tang G, Zhang X, Jiang J, Tian Q (2018) Haze removal method for natural restoration of images with sky. Neurocomputing 275:499–510

    Article  Google Scholar 

  27. Chao L, Wang M (2010) Removal of water scattering. In: Proceedings of the IEEE international conference on computing engineering and technology (ICCET), pp V2-35–V2-39

  28. Chiang JY, Chen YC (2012) Underwater image enhancement by wavelength compensation and dehazing. IEEE Trans Image Process 21(4):1756–1769

    Article  MathSciNet  MATH  Google Scholar 

  29. Galdran A, Pardo D, Picón A, Alvarez-Gila A (2015) Automatic red-channel underwater image restoration. J Vis Commun Image Represent 26:132–145

    Article  Google Scholar 

  30. Li C, Guo J, Chen S, Tang Y, Pang Y, Wang J (2016) Underwater image restoration based on minimum information loss principle and optical properties of underwater imaging. In: 2016 IEEE international conference on image processing (ICIP). IEEE, pp 1993–1997

  31. Oreifej O, Li X, Shah M (2013) Simultaneous video stabilization and moving object detection in turbulence. IEEE Trans Pattern Anal Mach Intell 35(2):450–462

    Article  Google Scholar 

  32. Gilles J, Alvarez F, Ferrante N, Fortman M, Tahir L, Tarter A, von Seeger A (2018) Detection of moving objects through turbulent media. Decomposition of oscillatory vs non-oscillatory spatio-temporal vector fields. Image Vis Comput 73:40–55

    Article  Google Scholar 

  33. Lee D, Kim G, Kim D, Myung H, Choi HT (2012) Vision-based object detection and tracking for autonomous navigation of underwater robots. Ocean Eng 48:59–68

    Article  Google Scholar 

  34. Edgington DR, Salamy KA, Risi M, Sherlock RE, Walther D, Koch C (2003) Automated event detection in underwater video. In: OCEANS 2003. Proceedings, vol 5. IEEE, pp P2749–P2753

  35. Rizzini DL, Kallasi F, Oleari F, Caselli S (2015) Investigation of vision-based underwater object detection with multiple datasets. Int J Adv Robot Syst 12(6):77–89

    Article  Google Scholar 

  36. Chuang MC, Hwang JN, Williams K (2016) A feature learning and object recognition framework for underwater fish images. IEEE Trans Image Process 25(4):1862–1872

    MathSciNet  MATH  Google Scholar 

  37. Zhu Y, Chang L, Dai J, Zheng H, Zheng B (2016) Automatic object detection and segmentation from underwater images via saliency-based region merging. In: OCEANS 2016-Shanghai. IEEE, pp 1–4

  38. Zhang Y, Yang Q (2017) An overview of multi-task learning. Natl Sci Rev 5(1):30–43

    Article  Google Scholar 

  39. Zhu Xiaofeng, Zhang Shichao, Rongyao Hu, Zhu Yonghua, Song Jingkuan (2018) Local and global structure preservation for robust unsupervised spectral feature selection. IEEE Trans Knowl Data Eng 30(3):517–529

    Article  Google Scholar 

  40. Zheng Wei, Zhu Xiaofeng, Zhu Yonghua, Rongyao Hu, Lei Cong (2017) Dynamic graph learning for spectral feature selection. Multimed Tools Appl. https://doi.org/10.1007/s11042-017-5272-y

    Article  Google Scholar 

  41. Elgammal A, Harwood D, Davis L (2000) Non-parametric model for background subtraction. In: European conference on computer vision. Springer, Berlin, pp 751–767

    Google Scholar 

  42. Fixed camera on coral reef. https://www.youtube.com/watch?v=kYZQaRS5Q2A. Accessed 6 Nov 2017

  43. Foggy Morning with Traffic. https://www.youtube.com/watch?v=ekh-BaoCLPU. Accessed 17 Dec 2016

  44. Heavy Fog Disrupts Traffic. https://www.youtube.com/watch?v=jde2I1PSW4Y. Accessed 5 Feb 2017

  45. Sony WR632 Traffic+fog fullzoom. https://www.youtube.com/watch?v=TBxr4R8-_ME. Accessed 9 Dec 2014

  46. Traffic congestion as heavy fog. https://www.youtube.com/watch?v=wwxhlFo_Nqw. Accessed 6 Feb 2017

  47. Static shot of street as people are walking and fog blows through. https://www.youtube.com/watch?v=CWNaPcbc1hE. Accessed 30 June 2015

  48. Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2):303–338

    Article  Google Scholar 

  49. Smeulders AW, Chu DM, Cucchiara R, Calderara S, Dehghan A, Shah M (2014) Visual tracking: an experimental survey. IEEE Trans Pattern Anal Mach Intell 36(7):1442–1468

    Article  Google Scholar 

  50. Xu M, Sun Q, Huang C, Shi J (2017) Object motion detection and data processing in large-scale particle image velocimetry. Intell Autom Soft Comput 23(4):653–660

    Article  Google Scholar 

  51. Elgammal A, Duraiswami R, Harwood D, Davis LS (2002) Background and foreground modeling using nonparametric kernel density estimation for visual surveillance. Proc IEEE 90(7):1151–1163

    Article  Google Scholar 

  52. Babacan SD, Pappas TN (2007) Spatiotemporal algorithm for background subtraction. In: IEEE international conference on acoustics, speech and signal processing. ICASSP 2007, vol 1. IEEE, , p I-1065

  53. Barnich O, Van Droogenbroeck M (2011) ViBe: a universal background subtraction algorithm for video sequences. IEEE Trans Image Process 20(6):1709–1724

    Article  MathSciNet  MATH  Google Scholar 

  54. Zhou X, Yang C, Yu W (2013) Moving object detection by detecting contiguous outliers in the low-rank representation. IEEE Trans Pattern Anal Mach Intell 35(3):597–610

    Article  Google Scholar 

  55. Tu Z, Abel A, Zhang L, Luo B, Hussain A (2016) A new spatio-temporal saliency-based video object segmentation. Cogn Comput 8(4):629–647

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Nos. 61501173, 61563036), the Fundamental Research Funds for the Central Universities (Nos. 2018B16514, 2017B01914), the Natural Science Foundation of Jiangsu Province (No. BK20150824), and the Jiangsu Overseas Scholar Program for University Prominent Young and Middle-aged Teachers and Presidents.

Author information

Authors and Affiliations

  1. College of Computer and Information, Hohai University, Nanjing, Jiangsu, China

    Zhe Chen, Xin Wang & Lizhong Xu

  2. School of Information Engineering, Nanchang Institute of Technology, Nanchang, Jiangxi, China

    Tanghuai Fan

Authors
  1. Zhe Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tanghuai Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lizhong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lizhong Xu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Z., Wang, X., Fan, T. et al. Color–depth multi-task learning for object detection in haze. Neural Comput & Applic 32, 6591–6599 (2020). https://doi.org/10.1007/s00521-018-3732-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-018-3732-6

Keywords

Search

Navigation