Skip to main content

Advertisement

SpringerLink
Log in

Fusion of thermal and visible cameras for the application of pedestrian detection

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

In this paper, we propose methods to calibrate visible and thermal cameras and register their images in the application of pedestrian detection. We calibrate the camera using a checkerboard pattern mounted on a heated rig. We implement the image registration using three different approaches. In the first approach, we use the camera calibration information to generate control points from the checkerboard pattern. These control points are then used to register the images. In the second approach, we generate trajectory points for image registration using an external illuminated object. In the third approach, we achieve the registration through face tracking without the aid of any external object. The particle swarm optimization algorithm performs the image registration using the generated control and trajectory points, observed in both the cameras. We demonstrate the advantages of fusing the thermal and visible camera within a pedestrian detection algorithm. We evaluate the proposed registration algorithms and perform a comparison with baseline algorithms, i.e. genetic and simulated annealing algorithms. Additionally, we also perform a detailed parameter evaluation of the particle swarm optimization algorithm. The experimental results demonstrate the accuracy of the proposed algorithm and the advantages of thermal-visible camera fusion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Bilodeau, G.A., Torabi, A., Morin, F.: Visible and infrared image registration using trajectories and composite foreground images. Image Vis. Comput. 29(1), 41–50 (2011)

    Article  Google Scholar 

  2. Bilodeau, G.A., Torabi, A., St-Charles, P.L., Riahi, D.: Thermalvisible registration of human silhouettes: a similarity measure performance evaluation. Infrared Phys. Technol. 64, 79–86 (2014)

    Article  Google Scholar 

  3. Bouguet, J.: Camera calibration toolbox for matlab. https://www.vision.caltech.edu/bouguetj/calib_doc/ (2016)

  4. Briers, M., Doucet, A., Maskell, S.: Smoothing algorithms for state-space models. Ann. Inst. Stat. Math. 62(1), 61–89 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  5. Chakravorty, T., Bilodeau, G., Granger, E.: Automatic image registration in infrared-visible videos using polygon vertices. In: CoRR (2014)

  6. Dana, K.J., Anandan, P.: Registration of visible and infrared images. In: SPIE (1993)

  7. Erkanli, S.: Fusion of visual and thermal images using genetic algorithms. Ph.D. Thesis (2011)

  8. FLIR (2015). http://www.flir.com

  9. Flitti, F., Collet, C., Slezak, E.: Image fusion based on pyramidal multiband multiresolution markovian analysis. Signal Image Video Process. 3(3), 275–289 (2008)

    Article  MATH  Google Scholar 

  10. Gade, R., Moeslund, T.B.: Thermal cameras and applications: a survey. Mach. Vis. Appl. 25(1), 245–262 (2014)

    Article  Google Scholar 

  11. Han, J., Pauwels, E.J., De Zeeuw, P.: Visible and infrared image registration in man-made environments employing hybrid visual features. Pattern Recognit. Lett. 34(1), 42–51 (2013)

    Article  Google Scholar 

  12. Hermosilla, G., Gallardo, F., Farias, G., Martin, C.S.: Fusion of visible and thermal descriptors using genetic algorithms for face recognition systems. Sensors 15(8), 17,944–17,962 (2015)

    Article  Google Scholar 

  13. IDS. https://en.ids-imaging.com/ (2015)

  14. Jiang, J., Zhang, X.: Visible and infrared image automatic registration algorithm using mutual information. In: Control and Decision Conference (2010)

  15. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: IEEE International Conference on Neural Networks (1995)

  16. Klimaszewski, J., Kondej, M., Kawecki, M., Putz, B.: Registration of infrared and visible images based on edge extraction and phase correlation approaches. Image Process. Commun. Chall. 4, 153–162 (2013)

  17. Leykin, A., Hammoud, R.: Pedestrian tracking by fusion of thermal-visible surveillance videos. Mach. Vis. Appl. 21(4), 587–595 (2008)

    Article  Google Scholar 

  18. Liu, Z., Laganière, R.: Context enhancement through infrared vision: a modified fusion scheme. Signal Image Video Process. 1(4), 293–301 (2007)

    Article  MATH  Google Scholar 

  19. Shah, P., Merchant, S.N., Desai, U.B.: Multifocus and multispectral image fusion based on pixel significance using multiresolution decomposition. Signal Image Video Process. 7(1), 95–109 (2013a)

    Article  Google Scholar 

  20. Shah, P., Reddy, B.C.S., Merchant, S.N., Desai, U.B.: Context enhancement to reveal a camouflaged target and to assist target localization by fusion of multispectral surveillance videos. Signal Image Video Process. 7(3), 537–552 (2013b)

    Article  Google Scholar 

  21. Shah, P., Srikanth, T.V., Merchant, S.N., Desai, U.B.: Multimodal image/video fusion rule using generalized pixel significance based on statistical properties of the neighborhood. Signal Image Video Process. 8(4), 723–738 (2014)

    Article  Google Scholar 

  22. Shi, Y., Eberhart, R.: A modified particle swarm optimizer. In: International Conference on Evolutionary Computation, pp 69–73 (1998)

  23. Sonn, S., Bilodeau, G.A., Galinier, P.: Fast and accurate registration of visible and infrared videos. In: Computer Vision and Pattern Recognition Workshops (2013)

  24. St-Laurent, L., Prvost, D., Maldague, X.: Fast and accurate calibration-based thermal/colour sensors registration. In: Quantitative Infrared Thermography (2010)

  25. Stolkin, R., Rees, D., Talha, M., Florescu, I.: Bayesian fusion of thermal and visible spectra camera data for region based tracking with rapid background adaptation. In: Multisensor Fusion and Integration for Intelligent Systems (2012)

  26. Urfalioglu, O.: Robust estimation of camera rotation, translation and focal length at high outlier rates. In: Proceedings of Canadian Conference on Computer and Robot Vision (2004)

  27. Ursine, W., Calado, F., Teixeira, G., Diniz, H., Silvino, S., de Andrade, R.: Thermal/visible autonomous stereo visio system calibration methodology for non-controlled environments. In: Quantitative Infrared Thermography (2012)

  28. Vidas, S., Lakemond, R., Denman, S., Fookes, C., Sridharan, S., Wark, T.: A mask-based approach for the geometric calibration of thermal-infrared cameras. IEEE Trans. Instrum. Meas. 61(6), 1625–1635 (2012)

    Article  Google Scholar 

  29. Yang, K., Chen, T., Xing, S., Li, J.: Infrared and visible image registration base on sift features. Key Eng. Mater. 500, 383–389 (2012)

    Article  Google Scholar 

  30. Yi, X., Wang, B., Fang, Y., Liu, S.: Registration of infrared and visible images based on the correlation of the edges. In: International Congress on Image and Signal Processing (2013)

  31. Zhao, J., Cheung, S.: Human segmentation by fusing visible-light and thermal imaginary. In: Computer Vision Workshops (2009)

Download references

Author information

Author notes

    Authors and Affiliations

    1. Toyota Technological Institute, Nagoya, Japan

      Vijay John, Shogo Tsuchizawa & Seiichi Mita

    2. University of British Columbia, Kelowna, Canada

      Zheng Liu

    Authors
    1. Vijay John
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Shogo Tsuchizawa
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Zheng Liu
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Seiichi Mita
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Vijay John.

    Additional information

    Vijay John and Shogo Tsuchizawa have equally contributed to this work.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    John, V., Tsuchizawa, S., Liu, Z. et al. Fusion of thermal and visible cameras for the application of pedestrian detection. SIViP 11, 517–524 (2017). https://doi.org/10.1007/s11760-016-0989-z

    Download citation

    • Received:

    • Revised:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11760-016-0989-z

    Keywords

    Search

    Navigation