Skip to main content

Advertisement

Springer Nature Link
Log in

Robust thermal infrared tracking via an adaptively multi-feature fusion model

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

When dealing with complex thermal infrared (TIR) tracking scenarios, the single category feature is not sufficient to portray the appearance of the target, which drastically affects the accuracy of the TIR target tracking method. In order to address these problems, we propose an adaptively multi-feature fusion model (AMFT) for the TIR tracking task. Specifically, our AMFT tracking method adaptively integrates hand-crafted features and deep convolutional neural network (CNN) features. In order to accurately locate the target position, it takes advantage of the complementarity between different features. Additionally, the model is updated using a simple but effective model update strategy to adapt to changes in the target during tracking. In addition, a simple but effective model update strategy is adopted to adapt the model to the changes of the target during the tracking process. We have shown through ablation studies that the adaptively multi-feature fusion model in our AMFT tracking method is very effective. Our AMFT tracker performs favorably on PTB-TIR and LSOTB-TIR benchmarks compared with state-of-the-art trackers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. He Y-J, Li M, Zhang J, Yao J-P (2015) Infrared target tracking via weighted correlation filter. Infrared Phys Technol 73:103–114

    Article  Google Scholar 

  2. Liu Q, Lu X, He Z, Zhang C, Chen W-S (2017) Deep convolutional neural networks for thermal infrared object tracking. Knowl Based Syst 134:189–198

    Article  Google Scholar 

  3. Wang Y, Wei X, Tang X, Wu J, Fang J (2022) Response map evaluation for RGBT tracking. Neural Comput Appl 34(7):5757–5769

    Article  Google Scholar 

  4. Gundogdu E, Koc A, Solmaz B, Hammoud RI, Aydin Alatan A (2016) Evaluation of feature channels for correlation-filter-based visual object tracking in infrared spectrum. In: CVPRW, IEEE, pp 24–32

  5. Lamberti F, Sanna A, Paravati G (2011) Improving robustness of infrared target tracking algorithms based on template matching. IEEE Trans Aerosp Electron Syst 47(2):1467–1480

    Article  Google Scholar 

  6. Chen J, Lin Y, Huang D, Zhang J (2020) Robust tracking algorithm for infrared target via correlation filter and particle filter. Infrared Phys Technol 111:103516103516

    Article  Google Scholar 

  7. He Y, Li M, Zhang J, Yao J (2015) Infrared target tracking based on robust low-rank sparse learning. IEEE Geosci Remote Sens Lett 13(2):232–236

    Article  Google Scholar 

  8. Yuan D, Chang X, Liu Q, Wang D, He Z (2021) Active learning for deep visual tracking. arXiv preprint arXiv:2110.13259

  9. Wang P, Sun M, Wang H, Li X, Yang Y (2020) Convolution operators for visual tracking based on spatial-temporal regularization. Neural Comput Appl 32(10):5339–5351

    Article  Google Scholar 

  10. Song X, Jin Z (2022) Robust label rectifying with consistent contrastive-learning for domain adaptive person re-identification. IEEE Trans Multimedia 24:3229–3239

    Article  Google Scholar 

  11. Shu X, Yang Y, Wu B (2021) A neighbor level set framework minimized with the split Bregman method for medical image segmentation. Signal Process 189:108293

    Article  Google Scholar 

  12. Li R, Zhang B, Kang D-J, Teng Z (2019) Deep attention network for person re-identification with multi-loss. Comput Electr Eng 79:106455

    Article  Google Scholar 

  13. Yuan D, Fan N, He Z (2020) Learning target-focusing convolutional regression model for visual object tracking. Knowl Based Syst 194:105526

    Article  Google Scholar 

  14. Shu X, Yang Y, Wu B (2021) Adaptive segmentation model for liver CT images based on neural network and level set method. Neurocomputing 453:438–452

    Article  Google Scholar 

  15. Song X, Jin Z (2022) Domain adaptive attention-based dropout for one-shot person re-identification. Int J Mach Learn Cybern 13(1):255–268

    Article  Google Scholar 

  16. Yan C, Chang X, Li Z, Guan W, Ge Z, Zhu L, Zheng Q (2021) Zeronas: differentiable generative adversarial networks search for zero-shot learning. IEEE Trans Pattern Anal Mach Intell 41:1–9

    Article  Google Scholar 

  17. Gao P, Ma Y, Song K, Li C, Wang F, Xiao L (2018) Large margin structured convolution operator for thermal infrared object tracking. In: ICPR, IEEE, pp 2380–2385

  18. Liu Q, Li X, He Z, Fan N, Yuan D, Liu W, Liang Y (2020) Multi-task driven feature models for thermal infrared tracking. In: AAAI, vol 34, AAAI, pp 11604–11611

  19. Zhang L, Gonzalez-Garcia A, Van De Weijer J, Danelljan M, Khan FS (2018) Synthetic data generation for end-to-end thermal infrared tracking. IEEE Trans Image Process 28(4):1837–1850

    Article  Google Scholar 

  20. Li X, Liu Q, Fan N, He Z, Wang H (2019) Hierarchical spatial-aware Siamese network for thermal infrared object tracking. Knowl Based Syst 166:71–81

    Article  Google Scholar 

  21. Li M, Peng L, Chen Y, Huang S, Qin F, Peng Z (2019) Mask sparse representation based on semantic features for thermal infrared target tracking. Remote Sens 11(17):1967

    Article  Google Scholar 

  22. Liu Q, He Z, Li X, Zheng Y (2019) PTB-TIR: a thermal infrared pedestrian tracking benchmark. IEEE Trans Multimedia 22(3):666–675

    Article  Google Scholar 

  23. Liu Q, Li X, He Z, et al (2020) LSOTB-TIR: a large-scale high-diversity thermal infrared object tracking benchmark. In: ACM MM, ACM, pp 3847–3856

  24. Li R, Zhang B, Teng Z, Fan J (2022) An end-to-end identity association network based on geometry refinement for multi-object tracking. Pattern Recogn 129:108738

    Article  Google Scholar 

  25. Marvasti-Zadeh SM, Ghanei-Yakhdan H, Kasaei S (2021) Efficient scale estimation methods using lightweight deep convolutional neural networks for visual tracking. Neural Comput Appl 33(14):8319–8334

    Article  Google Scholar 

  26. Yuan D, Kang W, He Z (2020) Robust visual tracking with correlation filters and metric learning. Knowl Based Syst 195:105697

    Article  Google Scholar 

  27. Dawoud A, Alam MS, Bal A, Loo C (2006) Target tracking in infrared imagery using weighted composite reference function-based decision fusion. IEEE Trans Image Process 15(2):404–410

    Article  Google Scholar 

  28. Yuan D, Chang X, Li Z, He Z (2021) Learning adaptive spatial-temporal context-aware correlation filters for UAV tracking. ACM Trans Multimed Comput Commun Appl 18(3):70:1-70:18

    Google Scholar 

  29. Yu T, Mo B, Liu F, Qi H, Liu Y (2019) Robust thermal infrared object tracking with continuous correlation filters and adaptive feature fusion. Infrared Phys Technol 98:69–81

    Article  Google Scholar 

  30. Li G, Peng M, Nai K, Li Z, Li K (2020) Multi-view correlation tracking with adaptive memory-improved update model. Neural Comput Appl 32(13):9047–9063

    Article  Google Scholar 

  31. Liu Q, Li X, He Z, Fan N, Yuan D, Wang H (2021) Learning deep multi-level similarity for thermal infrared object tracking. IEEE Trans Multimedia 23:2114–2126

    Article  Google Scholar 

  32. Qi Y, Zhang S, Qin L, Yao H, Huang Q, Lim J, Yang M-H (2016) Hedged deep tracking. In: CVPR, IEEE, pp 4303–4311

  33. Yuan D, Zhang X, Liu J, Li D (2019) A multiple feature fused model for visual object tracking via correlation filters. Multimedia Tools Appl 78(19):27271–27290

    Article  Google Scholar 

  34. Li X, Huang L, Wei Z, Nie J, Chen Z (2021) Adaptive multi-branch correlation filters for robust visual tracking. Neural Comput Appl 33(7):2889–2904

    Article  Google Scholar 

  35. Henriques JF, Caseiro R, Martins P, Batista J (2014) High-speed tracking with kernelized correlation filters. IEEE Trans Pattern Anal Mach Intell 37(3):583–596

    Article  Google Scholar 

  36. Kiani Galoogahi H, Fagg A, Lucey S (2017) Learning background-aware correlation filters for visual tracking. In: ICCV, IEEE, pp 1135–1143

  37. Yuan D, Shu X, He Z (2020) TRBACF: learning temporal regularized correlation filters for high performance online visual object tracking. J Vis Commun Image Rep 72:102882

    Article  Google Scholar 

  38. Danelljan M, Hager G, Shahbaz Khan F, Felsberg M (2015) Learning spatially regularized correlation filters for visual tracking. In: ICCV, IEEE, pp 4310–4318

  39. Bibi A, Mueller M, Ghanem B (2016) Target response adaptation for correlation filter tracking. In: ECCV, Springer, pp 419–433

  40. Yuan D, Li X, He Z, Liu Q, Lu S (2020) Visual object tracking with adaptive structural convolutional network. Knowl Based Syst 194:105554

    Article  Google Scholar 

  41. Yang K, Song H, Zhang K, Liu Q (2020) Hierarchical attentive Siamese network for real-time visual tracking. Neural Comput Appl 32(18):14335–14346

    Article  Google Scholar 

  42. Wang N, Song Y, Ma C, Zhou W, Liu W, Li H (2019) Unsupervised deep tracking. In: CVPR, IEEE, pp 1308–1317

  43. Bertinetto L, Valmadre J, Henriques JF, Vedaldi A, Torr PH (2016) Fully-convolutional Siamese networks for object tracking. In: ECCV, Springer, pp 850–865

  44. Valmadre J, Bertinetto L, Henriques J, Vedaldi A, Torr PH (2017) End-to-end representation learning for correlation filter based tracking. In: CVPR, IEEE, pp 2805–2813

  45. Dong X, Shen J (2018) Triplet loss in Siamese network for object tracking. In: ECCV, Springer, pp 459–474

  46. Song Y, Ma C, Gong L, Zhang J, Lau RW, Yang M-H (2017) CREST: convolutional residual learning for visual tracking. In: ICCV, IEEE, pp 2574–2583

  47. Li R, Zhang B, Teng Z, Fan J (2021) A divide-and-unite deep network for person re-identification. Appl Intell 51(3):1479–1491

    Article  Google Scholar 

  48. Yuan D, Shu X, Liu Q, He Z (2022) Structural target-aware model for thermal infrared tracking. Neurocomputing 491:44–56

    Article  Google Scholar 

  49. Ma C, Huang J-B, Yang X, Yang M-H (2015) Hierarchical convolutional features for visual tracking. In: ICCV, IEEE, pp 3074–3082

  50. Li M, Cai W, Verspoor K, Pan S, Liang X, Chang X (2022) Cross-modal clinical graph transformer for ophthalmic report generation. In: CVPR, pp 20656–20665

  51. Elayaperumal D, Joo YH (2021) Robust visual object tracking using context-based spatial variation via multi-feature fusion. Inf Sci 577:467–482

    Article  Google Scholar 

  52. Li M, Huang P-Y, Chang X, Hu J, Yang Y, Hauptmann A (2022) Video pivoting unsupervised multi-modal machine translation. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2022.3181116

    Article  Google Scholar 

  53. Zhang L, Danelljan M, Onzalez-Garcia A, van de Weijer J, Shahbaz Khan F (2019) Multi-modal fusion for end-to-end rgb-t tracking. In: ICCVW, IEEE, pp 2252–2261

  54. Li C, Lu A, Zheng A, Tu Z, Tang J (2019) Multi-adapter RGBT tracking. In: ICCVW, IEEE, pp 2262–2270

  55. Wang F, Vemuri BC, Rangarajan A (2006) Groupwise point pattern registration using a novel CDF-based Jensen–Shannon divergence. In: CVPR, IEEE, pp 1283–288

  56. Sutter T, Daunhawer I, Vogt JE (2020) Multimodal generative learning utilizing Jensen–Shannon divergence. In: NeurIPS, Curran, pp 6100–6110

  57. Li X, Liu Q, He Z, Wang H, Zhang C, Chen W-S (2016) A multi-view model for visual tracking via correlation filters. Knowl Based Syst 113:88–99

    Article  Google Scholar 

  58. Li X, Ma C, Wu B, He Z, Yang M-H (2019) Target-aware deep tracking. In: CVPR, IEEE, pp 1369–1378

  59. Song Y, Ma C, Wu X, Gong L, Bao L, Zuo W, Shen C, Lau RW, Yang M-H (2018) Vital: visual tracking via adversarial learning. In: CVPR, IEEE, pp 8990–8999

  60. Xu T, Feng Z-H, Wu X-J, Kittler J (2019) Joint group feature selection and discriminative filter learning for robust visual object tracking. In: ICCV, IEEE, pp 7950–7960

  61. Nam H, Han B (2016) Learning multi-domain convolutional neural networks for visual tracking. In: CVPR, IEEE, pp 4293–4302

  62. Bertinetto L, Valmadre J, Golodetz S, Miksik O, Torr PH (2016) Staple: complementary learners for real-time tracking. In: CVPR, IEEE, pp 1401–1409

  63. Wang N, Zhou W, Tian Q, Hong R, Wang M, Li H (2018) Multi-cue correlation filters for robust visual tracking. In: CVPR, IEEE, pp 4844–4853

  64. Danelljan M, Shahbaz Khan F, Felsberg M, Van de Weijer J (2014) Adaptive color attributes for real-time visual tracking. In: CVPR, IEEE, pp 1090–1097

  65. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR, IEEE, pp 770–778

Download references

Acknowledgements

This study was supported by the National Natural Science Foundation of China (Grant Nos. 62202362, 61672183, 62172126), by the China Postdoctoral Science Foundation (Grant No. 2022TQ0247), by the Natural Science Foundation of Chongqing (Grant No.ncamc2022-msxm03), Science Foundation of The Chongqing Education Commission (Grant No.KJZD-K202200501), Foundation Project of Chongqing Normal University (Grant No.21XLB024) by the Special Research project on COVID-19 Prevention and Control of Guangdong Province (Grant No. 2020KZDZDX1227), by the Shenzhen Research Council (Grant No. JCYJ20210324120202006), by the Fundamental Research Funds for the Central Universities (Grant No. XJS222503), and by the Foundation Project of Guangzhou Institute of Technology, Xidian University (Grant No. 01131002).

Author information

Authors and Affiliations

  1. Guangzhou Institute of Technology, Xidian University, Guangzhou, 510555, China

    Di Yuan

  2. School of Science, Harbin Institute of Technology, Shenzhen, 518055, China

    Xiu Shu & Xinming Zhang

  3. National Center for Applied Mathematics in Chongqing, Chongqing Normal University, Chongqing, 401331, China

    Qiao Liu

  4. School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518055, China

    Zhenyu He

Authors
  1. Di Yuan
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Xiu Shu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Qiao Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Xinming Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Zhenyu He
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Di Yuan.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yuan, D., Shu, X., Liu, Q. et al. Robust thermal infrared tracking via an adaptively multi-feature fusion model. Neural Comput & Applic 35, 3423–3434 (2023). https://doi.org/10.1007/s00521-022-07867-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07867-1

Keywords