Skip to main content
SpringerLink
Log in

An IR and visible image sequence automatic registration method based on optical flow

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

IR–visible camera registration is required for multi-sensor fusion and cooperative processing. Image sequences can provide motion information, which is useful for sequence registration. The existing methods mainly focus on registration using moving objects which are observed by both cameras. However, accurate motion feature extraction for a whole moving object is difficult, because of the complex environment and different imaging mechanism of two sensors. To overcome this problem, we use motion features associated with single pixels in the two image sequences to carry out automatic registration. A normalized optical flow time sequence for each image pixel is constructed. The matching of pixels between the IR image and the visible light image is carried out using a fast similarity measurement and a three stage correspondence selection method. Finally cascaded random sample consensus is adopted to remove outlying matches, and least-square method and Levenberg–Marquardt method are used to estimate the transformation from the IR image to the visible image. The effectiveness of our method is demonstrated using several real datasets and simulated datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Lu, X., Zhang, S., Su, H., Chen, Y.: Mutual information-based multimodal image registration using a novel joint histogram estimation. Comput. Med. Imaging Graph. 32(3), 202–209 (2008)

    Article  Google Scholar 

  2. Luan, H.X., Qi, F.H., Xue, Z.H.: Multimodality image registration by maximization of quantitative–qualitative measure of mutual information. Pattern Recognit. 41(1), 285–298 (2008)

    Article  MATH  Google Scholar 

  3. Istenic, R., Heric, D., Ribaric, S.: Thermal and visual image registration in hough parameter space. In: Proceedings of the 14th International Workshop on Systems, Signals and Image Processing and 6th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services, pp. 106–109 (2007)

  4. Coiras, E., Santamaria, J., Miravet, C.: Segment-based registration technique for visual-infrared images. Opt. Eng. 39(1), 282–289 (2000)

    Google Scholar 

  5. Lee, H.J., Kim, S.Y., Lee, D.: Robust CCD and IR image registration using gradient-based statistical information. IEEE Signal Process. Lett. 17(4), 347–350 (2010)

    Google Scholar 

  6. Kim, Y.S., Lee, J.H., Ra, J.B.: Multi-sensor image registration based on intensity and edge orientation information. Pattern Recognit. 41(11), 3356–3365 (2008)

    Article  MATH  Google Scholar 

  7. Hrkac, T., Kalafatic, Z., Krapac, J.: Infrared-visual image registration based on corners and hausdorff distance. Lect. Notes Comput. Sci. 4522, 383–392 (2007)

    Article  Google Scholar 

  8. Irani, M., Anandan, P.: Robust multi-sensor image alignment. In: Proceedings of International Conference on Computer Vision, pp. 959–966 (1998)

  9. Lee, L., Romano, R., Stein, G.P.: Monitoring activities from multiple video streams: establishing a common coordinate frame. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 758–767 (2000)

    Article  Google Scholar 

  10. Caspi, Y., Simakov, D., Irani, M.: Feature-based sequence-to-sequence matching. Int. J. Comput. Vis. 68(1), 53–64 (2006)

    Article  Google Scholar 

  11. Ju, H., Bhanu, B.: Fusion of color and infrared video for moving human detection. Pattern Recognit. 40(6), 1771–1784 (2007)

    Article  MATH  Google Scholar 

  12. Verstockt, S., Poppe, C., Potter, D.P., Hollemeersch, C.: Silhouette coverage analysis for multi-modal video surveillance. PIERS Online 7(5), 401–405 (2011)

    Google Scholar 

  13. Torabi, A., Masse, G., Bilodeau, G-A.: Feedback scheme for thermal-visible video registration, sensor fusion, and people tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition-Workshop, pp. 15–22 (2010)

  14. Bilodeau, A.G., Torabi, A., Morin, F.: Visible and infrared image registration using trajectories and composite foreground images. Image Vis. Comput. 29(1), 41–50 (2011)

    Article  Google Scholar 

  15. Levenberg, K.: A method for the solution of certain nonlinear problems in least squares. Q. Appl. Math. 2(2), 164–168 (1994)

    Google Scholar 

  16. Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear parameters. SIAM J. Appl. Math. 11(2), 431–441 (1963)

    Article  MathSciNet  MATH  Google Scholar 

  17. Bouguet, J.Y.: Pyramidal implementation of the Lucas Kanade feature tracker: description of the algorithm. Technical report, Intel Corporation, Microprocessor Research Labs 2000, OpenCV documentation (2001)

  18. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  19. Davis, J., Sharma, V.: Background-subtraction using contour-based fusion of thermal and visible imagery. Comput. Vis. Image Underst. 106(23), 162–182 (2007)

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Nos. 60872145, 60903126), China Postdoctoral Special Science Foundation (No. 201003685), and China  Post doctoral Science Foundation (No. 20090451397).

Author information

Authors and Affiliations

  1. School of Computer Science, Northwestern Polytechnical University, Xi’an, 710072, China

    Yanning Zhang, Xiuwei Zhang & Rui Yu

  2. Department of Computer Science and Information Systems, Birkbeck College, University of London, London, WC1E 7HX, UK

    Xiuwei Zhang & Stephen J. Maybank

Authors
  1. Yanning Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiuwei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephen J. Maybank
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rui Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiuwei Zhang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, Y., Zhang, X., Maybank, S.J. et al. An IR and visible image sequence automatic registration method based on optical flow. Machine Vision and Applications 24, 947–958 (2013). https://doi.org/10.1007/s00138-012-0465-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-012-0465-x

Keywords

Search

Navigation