Skip to main content
SpringerLink
Log in

Fusion of thermal infrared and visible spectra for robust moving object detection

  • Survey
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

The detection of moving objects is a crucial step for many video surveillance applications whether using a visible camera (VIS) or an infrared (IR) one. In order to profit from both types, several fusion methods were proposed in the literature: low-level fusion, medium-level fusion and high-level fusion. The first one is the most used for moving objects’ detection in IR and VIS spectra. In this paper, we present an overview of the different moving object detection methods in IR and VIS spectra and a state of the art of the low-level fusion techniques. Moreover, we propose a new method for moving object detection using low-level fusion of IR and VIS spectra. In order to evaluate quantitatively and qualitatively our proposed method, three series of experiments were carried out using two well-known datasets namely “OSU Color-Thermal Database” and “INO-Database”; the results of these evaluations show promising results and demonstrate the effectiveness of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. http://vcipl-okstate.org/pbvs/bench/.

  2. http://www.ino.ca/en/video-analytics-dataset/.

References

  1. Pavlidis I, Morellas V, Tsiamyrtzis P, Harp S (2001) Urban surveillance systems: from the laboratory to the commercial world. Proc IEEE 89(10):1478–1497

    Article  Google Scholar 

  2. Zin TT, Takahashi H, Toriu T, Hama H (2011) Fusion of infrared and visible images for robust person detection. In: Ukimura O (ed) Image fusion. InTech, Rijeka

    Google Scholar 

  3. Conaire CO, Cooke E, O’Connor N, Murphy N, Smearson A (2005) Background modelling in infrared and visible spectrum video for people tracking. In: Proceedings of international conference on computer vision and pattern recognition, San Diego, California, pp 20–20

  4. Prajapati D, Galiyawala HJ (2015) A review on moving object detection and tracking. Int J Comput Appl 5(3):168–175

    Google Scholar 

  5. Joshi KA, Thakore DG (2012) A survey on moving object detection and tracking in video surveillance system. Int J Soft Comput Eng 2(3):44–48

    Google Scholar 

  6. Xu Y, Dong J, Zhang B, Xu D (2016) Background modeling methods in video analysis: a review and comparative evaluation. CAAI Trans Intell Technol 1(1):43–60

    Article  Google Scholar 

  7. Cheng YH, Wang J (2014) A motion image detection method based on the inter-frame difference method. Appl Mech Mater 490–491:1283–1286

    Google Scholar 

  8. Arvanitidou MG, Tok M, Glantz A, Krutz A, Sikora T (2013) Motion-based object segmentation using hysteresis and bidirectional inter-frame change detection in sequences with moving camera. Image Commun J 28(10):1420–1434

    Google Scholar 

  9. Zhen Y, Yanping C (2009) A real-time motion detection algorithm for traffic monitoring systems based on consecutive temporal difference. In: Proceedings of 7th Asian control conference, Hong Kong, pp 1594–1599

  10. Xin W, Gaolue L (2011) Fusion algorithm for infrared–visual image sequences. In: Proceedings of the 6th International Conference on Image and Graphics, Hefei, Anhui, pp 244–248

  11. Fernandez-Caballero A, Castillo JC, Martinez-Cantos J, Martinez-Tomas R (2010) Optical flow or image subtraction in human detection from infrared camera on mobile robot. Robot Auton Syst 58(12):1273–1281

    Article  Google Scholar 

  12. Jain R, Nagel H-H (1979) On the analysis of accumulative difference pictures from image sequences of real world scenes. IEEE Trans Pattern Anal Mach Intell 1(2):206–214

    Article  Google Scholar 

  13. Lillestrand RL (2006) Techniques for change detection. IEEE Trans Comput C–21(7):654–659

    Google Scholar 

  14. Asli RN, Zavaraki MM (2016) Fast-optimized object detection in dynamic scenes using efficient background weighting. Int J Hybrid Inf Technol 9(3):11–22

    Article  Google Scholar 

  15. Pang Y, Ye L, Li X, Pan J (2015) Moving object detection in video using saliency map and subspace learning. In: IEEE Transactions on Circuits Systems for Video Technology, pp 4321–4330

  16. Hou AL, Guo JL, Wang CJ, Wu L, Li F (2013) Abnormal behavior recognition based on trajectory feature and regional optical flow. In: Proceedings of the 7th international conference on image and graphics, Qingdao, pp 643–649

  17. Jian-Ping T, Xiao-lan L, Jun L (2016) Moving object detection and identification method based on vision. Int J Secur Appl 10(3):101–110

    Google Scholar 

  18. Ke H (2016) Moving object detection research based on background image set and sparse analysis. J Softw Eng 10(1):66–77

    Article  Google Scholar 

  19. Akula A, Khanna N, Ghosh R, Kumar S, Das A, Sardana HK (2014) Adaptive contour-based statistical background subtraction method for moving target detection in infrared video sequences. Infrared Phys Technol 63:103–109

    Article  Google Scholar 

  20. Chen BW, Liu SL (2014) Infrared target detection based on temporal–spatial domain fusion. Adv Mater Res 1044–1045:1186–1189

    Article  Google Scholar 

  21. Bondzulic B, Belgrade MA, Petrovic V (2008) Multisensor background extraction and updating for moving target detection. In: Proceedings of the 11th international conference on information fusion, Cologne, pp 1–8

  22. Goubet E, Katz J, Porikli F (2006) Pedestrian tracking using thermal infrared imaging. In: Proceedings of SPIE, vol 62062, pp 62062C–62062C12

    Google Scholar 

  23. Hariyono J, Hoang V-D, Jo K-H (2014) Moving object localization using optical flow for pedestrian detection from a moving vehicle. Sci World J 2014:1–8

    Article  Google Scholar 

  24. Pathirana PN, Lim AEK, Carminati J, Premaratne M (2007) Simultaneous estimation of optical flow and object state, a modified approach to optical flow calculation. In: Proceedings of IEEE international conference on networking, sensing and control, London, UK, pp 634–638

  25. Qi Y, An G (2011) Infrared moving targets detection based on optical flow estimation. In: Proceedings of the international conference on computer science and network technology, China, pp 2452–2455

  26. Brehar R, Nedevschi S (2014) Pedestrian detection in infrared images using HOG, LBP, gradient magnitude and intensity feature channels. In: Proceedings of IEEE 17th international conference on intelligent transportation systems, Qingdao, pp 1669–1674

  27. Gilmore ET, Ugbome C, Kim C (2011) An IR-based pedestrian detection system implemented with matlab-equipped laptop and low-cost microcontroller. Int J Comput Sci Inf Technol 3(5):79–87

    Google Scholar 

  28. Kancharla T, Kharade P, Gindi S, Kutty K, Vaidya VG (2011) Edge based segmentation for pedestrian detection using NIR camera. In: Proceedings of the international conference on image information processing, Himachal Pradesh, pp 1–6

  29. Olmeda D, Hilario C, Escalera A, Armingol JM (2008) Pedestrian detection and tracking based on far infrared visual information. In: Proceedings of the 10th international conference on advanced concepts for intelligent vision systems, France, pp 958–969

  30. Bertozzi M, Broggi A, Felisa M, Vezzoni G, Del Rose M (2006) Low-level pedestrian detection by means of visible and far infra-red tetra-vision. In: Proceedings of the IEEE intelligent vehicles symposium, Tokyo, pp 231–236

  31. Torresan H, Turgeon B, Ibarra-Castanedo C, Hebert P, Maldague X (2004) Advanced surveillance systems: combining video and thermal imagery for pedestrian detection. In: Proceedings of SPIE Thermosense XXVI, Vol 5405 of SPIE

  32. Tribaldos P, Serrano-Cuerda J, Lopez MT, Fernandez-Caballero A, Lopez-Sastre RJ (2013) People detection in color and infrared video using HOG and linear SVM. In: Proceedings of the 5th international work-conference on the natural and artificial computation in engineering and medical applications, Berlin, Heidelberg, pp 179–189

  33. Olmeda D, Escalera A, Armingol JM (2012) Contrast invariant features for human detection in far infrared images. In: Proceedings of IEEE intelligent vehicles symposium (IV), Alcala de Henares, pp 117–122

  34. Zin TT, Tin P, Hama H (2011) Pedestrian detection based on hybrid features using near infrared images. Int J Innov Comput Inf Control 7(8):5015–5025

    Google Scholar 

  35. Wang X, Han TX, Yan S (2009) An hog-lbp human detector with partial occlusion handling. In: Proceedings of IEEE 12th international conference on computer vision, Kyoto, pp 32–39

  36. Schwartz W, Kembhavi A, Harwood D, Davis L (2009) Human detection using partial least squares analysis. In: Proceedings of IEEE international conference on computer vision, Kyoto, pp 24–31

  37. Bertozzi M, Broggi A, Caraffi C, Del Rose M, Felisa M, Vezzoni G (2007) Pedestrian detection by means of far-infrared stereo vision. J Comput Vis Image Underst 106(2–3):194–204

    Article  Google Scholar 

  38. Dai C, Zheng Y, Li X (2007) Pedestrian detection and tracking in infrared imagery using shape and appearance. J Comput Vis Image Underst 106(2–3):288–299

    Article  Google Scholar 

  39. Bertozzi M, Broggi A, Hilario Gomez C, Fedriga RI, Vezzoni G, Del Rose M (2007) Pedestrian detection in far infrared images based on the use of probabilistic templates. In: Proceedings of IEEE symposium on intelligent vehicle, Istanbul, pp 327–332

  40. Buch N, Cracknell M, Orwell J, Velastin SA (2009) Vehicle localisation and classification in urban CCTV steams. 16th World Congress and exhibition on intelligent transport systems and services. Stockholm, Sweden, pp 1–8

  41. Leibe B, Leonardis A, Schiele B (2004) Combined object categorization and segmentation with an implicit shape model. Workshop on statistical learning in computer vision, Prague, Czech Republic, pp 1732

  42. Lin Z, Davis LS (2008) A pose-invariant descriptor for human detection and segmentation. In: European conference on computer vision, Berlin, Heidelberg, pp 423–436

  43. Meis M, Oberlander U, Ritter W (2004) Reinforcing the reliability of pedestrian detection in far-infrared sensing. In: Intelligent Vehicles Symposium, pp 779–783

  44. Cong DNT, Khoudour L, Achard C, Phothisane P (2009) People re-identification by means of a camera network using a graph-based approach. In: Conference on Machine Vision Applications, Yokohama, Japan, pp 152–155

  45. Fei M, Li J, Liu H (2015) Visual tracking based on improved foreground detection and perceptual hashing. Neurocomputing 152:413–428

    Article  Google Scholar 

  46. Kushwaha AKS, Srivastava S, Srivastava R (2016) Multi-view human activity recognition based on silhouette and uniform rotation invariant local binary patterns. Multimed Syst J 1–17. doi:10.1007/s00530-016-0505-x

  47. Tewary S, Akula A, Ghosh R, Kumar S, Sardana HK (2014) Hybrid multi-resolution detection of moving targets in infrared imagery. Infrared Phys Technol 67:173–183

    Article  Google Scholar 

  48. Yin J, Liu L, Li H, Liu Q (2016) The infrared moving object detection and security detection related algorithms based on W4 and frame difference. Infrared Phys Technol 77:302–315

    Article  Google Scholar 

  49. Lu N, Wang J, Wu QH, Yang L (2008) An improved motion detection method for real-time surveillance. Int J Comput Sci 35(1):119–128

    Google Scholar 

  50. Aptean AD (2011) Contributions to the information fusion: application to obstacle recognition in visible and infrared images, PhD thesis, National Institute of Applied Sciences of Rouen, France and Technical University of Cluj-Napoca, Romnia

  51. Wang J, Liang J, Hu H, Li Y, Feng B (2007) Performance evaluation of infrared and visible image fusion algorithms for face recognition. In: Proceedings of the international conference on intelligent systems and knowledge, Engineering, pp 1–8

  52. Davis JW, Sharma V (2007) Background-subtraction using contour-based fusion of thermal and visible imagery. Comput Vis Image Underst 106(2–3):162–182

    Article  Google Scholar 

  53. Xiao G, Wei K, Jing ZL (2008) Improved dynamic image fusion scheme for infrared and visible sequence based on image fusion system. In: Proceedings of the 11th international conference on information fusion, Cologne, pp 1745–1750

  54. Lewis JJ, O’Callaghan RJ, Nikolov SG, Bull DR, Canagarajah CN (2007) Pixel and region based image fusion with complex wavelets. J Inf Fusion 8(2):119–130

    Article  Google Scholar 

  55. Lanir Y (2005) Comparing multispectral image fusion methods for a target detection task. Thesis submitted in partial fulfillment of the requirements for the M.Sc Degree

  56. Yang B, Zhong-liang J, Hai-tao Z (2010) Review of pixel-level image fusion. J Shanghai Jiaotong Univ 15(1):6–12

    Article  Google Scholar 

  57. Pop S (2008) Modle de fusion et diffusion par quations aux drives partielles: application la sismique azimutale, PhD thesis, Bordeaux I University, France

  58. Blum RS, Xue Z, Zhang Z (2006) An overview of image fusion. In: Liu Z, Blum RS (eds) Multi-sensor image fusion and its applications. CRC Press, pp 1–36

  59. Li M, Dong Y (2013) Review on technology of pixel-level image fusion. In: Proceedings of the international conference on measurement, information and control, Harbin, pp 341–344

  60. Jagalingam P, Hegde AV (2014) Pixel level image fusion a review on various techniques. In: Proceedings of the 3rd world conference on applied sciences, engineering and technology, Kathmandu, Nepal, pp 521–528

  61. Yang B, Jing Z-liang, Zhao H-tao (2010) Review of pixel-level image fusion. J Shanghai Jiaotong Univ Sci 15(1):6–12

    Article  Google Scholar 

  62. Amro I, Mateos J, Vega M, Molina R, Katsaggelos AK (2011) A survey of classical methods and new trends in pansharping of multispectral images. EURASIP J Adv Signal Process 2011(1):1–22

    Article  Google Scholar 

  63. Singh S, Gyaourova A, Bebis G, Pavlidis I (2004) Infrared and visible image fusion for face recognition. Proc SPIE 5404:585–596

    Article  MATH  Google Scholar 

  64. Desale RP, Verma SV (2013) Study and analysis of PCA, DCT & DWT based image fusion techniques. In: Proceedings of IEEE international conference on signal processing image processing and pattern recognition, Coimbatore, pp 66–69

  65. Haghighat MBA, Aghagolzadeh A, Seyedarabi H (2010) Real-time fusion of multi-focus images for visual sensor networks. In: Proceedings of the 6th IEEE Iranian conference on machine vision and image processing, Isfahan, pp 1–6

  66. Sadjadi F (2005) Comparative image fusion analysais. In: IEEE Computer Society Conference on computer vision and pattern recognition—workshops, vol 8, No (8), pp 25–25

  67. Hammami M, Jarraya SK, Ben-Abdallah H (2013) On line background modeling for moving object segmentation in dynamic scene. Multimed Tools Appl 63(3):899–926

    Article  Google Scholar 

  68. Zhang E, Zhang Y (2009) F-Measure. In: Liu L, Özsu MT (eds) Encyclopedia of database systems. Springer, New York

    Google Scholar 

  69. Kim K, Chalidabhongse T, Harwood D, Davis L (2005) Real-time foreground-background segmentation using codebook model. Real Time Imaging J 11(3):167256

    Article  Google Scholar 

  70. Parag T (2014) Enforcing label and intensity consistency for IR target detection. In CoRR abs, pp 1–21  

  71. Mouats T, Aouf N (2014) Fusion of thermal and visible images for day/night moving objects detection. Sensor Signal Processing for Defence (SSPD), Edinburgh, pp 1–5

  72. Mangale S, Khambete M (2016) Camouflaged target detection and tracking using thermal infrared and visible spectrum imaging, Intelligent Systems Technologies and Applications, pp 193–207

Download references

Author information

Authors and Affiliations

  1. MIRACL-FS, Sfax University, Road Sokra Km 3 BP 802, 3018, Sfax, Tunisia

    Emna Fendri & Mohamed Hammami

  2. MIRACL-ISIMS, Sfax University, Sakiet Ezzit, Sfax, Tunisia

    Rania Rebai Boukhriss

Authors
  1. Emna Fendri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rania Rebai Boukhriss
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohamed Hammami
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rania Rebai Boukhriss.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fendri, E., Boukhriss, R.R. & Hammami, M. Fusion of thermal infrared and visible spectra for robust moving object detection. Pattern Anal Applic 20, 907–926 (2017). https://doi.org/10.1007/s10044-017-0621-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-017-0621-z

Keywords

Search

Navigation