Skip to main content
SpringerLink
Log in

A Siamese network-based tracking framework for hyperspectral video

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

With the rapid development of hyperspectral imaging techniques, hyperspectral video visual tracking comes to a breakthrough because its abundant material-based spectral information has strong discrimination ability in complex background. Most existing hyperspectral trackers use hand-craft features to represent the appearances of targets, but their performances are limited for the lack of semantic information in those low-level features. Despite the successes of deep networks on color video, the limited training samples bring difficulties to train a deep learning model-based hyperspectral tracker. To handle well with above problems, we present a novel deep hyperspectral tracker based on Siamese network (SiamHT). In our proposed method, heterogeneous encoder–decoder (HED) and spectral semantic representation (SSR) modules are designed to extract the spatial and spectral semantic features, respectively. After that parameters in HED and SSR modules are learned with a designed two-stage training strategy. Finally, the well-learned spatial and spectral semantic representations are fused to estimate the state of a target. Extensive comparison experiments on hyperspectral object tracking dataset are performed to prove the robustness of our method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Li P, Wang D et al (2018) Deep visual tracking: review and experimental comparison. Pattern Recognit 76:323–338

    Article  Google Scholar 

  2. Wang N, Shi J, Yeung D et al (2015) Understanding and diagnosing visual tracking systems. In: IEEE international conference on computer vision, pp 3101–3109

  3. Zhang T, Xu C, Yang MH (2019) Robust structural sparse tracking. IEEE Trans Pattern Anal Mach Intell 41(2):473–486

    Article  Google Scholar 

  4. Li B, Wu W, Wang Q, Zhang F, Xing J, Yan J (2019) SiamRPN++: evolution of siamese visual tracking with very deep networks. In: IEEE conference on computer vision and pattern recognition, pp 4277–4286

  5. Huang H, Shi G et al (2020) Feature learning using spatial-spectral hypergraph discriminant analysis for hyperspectral image. IEEE Trans Cybern 50(6):2604–2616

    Article  Google Scholar 

  6. Zhang L, Zhang L, Du B et al (2019) Hyperspectral image unsupervised classification by robust manifold matrix factorization. Inf Sci 485:154–169

    Article  MathSciNet  MATH  Google Scholar 

  7. Xiong F, Zhou J, Qian Y (2020) Material based object tracking in hyperspectral videos. IEEE Trans Image Process 20:3719–3733

    Article  MATH  Google Scholar 

  8. Qian K, Zhou J, Xiong F, Zhou H (2018) Object tracking in hyperspectral videos with convolutional features and kernelized correlation filter. In: International conference on smart multimedia, pp 308–319

  9. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations

  10. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition, pp 770–778

  11. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: IEEE conference on computer vision and pattern recognition, pp 2818–2826

  12. Voigtlaender P, Luiten J, Torr P, Leibe B (2020) Siam R-CNN: visual tracking by re-detection. In: IEEE conference on computer vision and pattern recognition, pp 6577–6587

  13. Yu Y, Xiong Y, Huang W, Scott M (2020) Deformable siamese attention networks for visual object tracking. In: IEEE conference on computer vision and pattern recognition, pp 6727–6736

  14. Bolme D, Beveridge JR, Draper B, Lui Y (2010) Visual object tracking using adaptive correlation filters. In: IEEE conference on computer vision and pattern recognition, pp 2544–2550

  15. Henriques J, Caseiro R, Martins P, Batista J (2012) Exploiting the circulant structure of tracking-by-detection with kernels. In: European conference on computer vision, pp 702–715

  16. Henriques J et al (2015) High-speed tracking with kernelized correlation filters. IEEE Trans Pattern Anal Mach Intell 37(3):583–596

    Article  Google Scholar 

  17. Danelljan M, Khan F, Felsberg M, Weijer J (2014) Adaptive color attributes for real-time visual tracking. In: IEEE conference on computer vision and pattern recognition, pp 1090–1097

  18. Bertinetto L, Valmadre J, Golodetz S, Miksik O, Torr PHS (2016) Staple: Complementary learners for real-time tracking. In: IEEE conference on computer vision and pattern recognition, pp 1401–1409

  19. Li Y, Zhu J (2014) A scale adaptive kernel correlation filter tracker with feature integration. In: European conference on computer vision, pp 254–265

  20. Danelljan M, Hager G, Khan F, Felsberg M (2017) Discriminative scale space tracking. IEEE Trans Pattern Anal Mach Intell 39(8):1561–1575

    Article  Google Scholar 

  21. Danelljan M et al. (2015) Learning spatially regularized correlation filters for visual tracking. In: IEEE international conference on computer vision, pp 4310–4318

  22. Galoogahi H, Fagg A, Lucey S (2017) Learning background-aware correlation filters for visual tracking. In: IEEE international conference on computer vision, pp 1135–1143

  23. Li F, Tian C, Zuo W, Zhang L, Yang MH (2018) Learning spatial-temporal regularized correlation filters for visual tracking. In: IEEE conference on computer vision and pattern recognition, pp 4904–4913

  24. Xu T, Feng Z et al (2019) Learning adaptive discriminative correlation filters via temporal consistency preserving spatial feature selection for robust visual tracking. IEEE Trans Image Process 28(11):5596–5609

    Article  MathSciNet  MATH  Google Scholar 

  25. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends in Mach Learn 3(1):1–122

    Article  MATH  Google Scholar 

  26. Dai K, Wang D, Lu H et al. (2019) Visual tracking via adaptive spatially-regularized correlation filters. In: IEEE conference on computer vision and pattern recognition, pp 4665–4674

  27. Xu T, Feng Z et al. (2019) Joint group feature selection and discriminative filter learning for robust visual object tracking. In: IEEE international conference on computer vision, pp 6181–6190

  28. Bertinetto L, Valmadre J, Henriques J, Vedaldi A, Torr P (2016) Fully-Convolutional Siamese networks for object tracking. In: European conference on computer vision, pp 850–865

  29. He A, Luo C, Tian X, Zeng W (2018) A twofold Siamese network for real-time object tracking. In: IEEE conference on computer vision and pattern recognition, pp 4834–4843

  30. Wang Q, Teng Z, Xing J, Gao J, Hu W, Maybank S (2018) Learning Attentions: Residual attentional Siamese network for high performance online visual tracking. In: IEEE conference on computer vision and pattern recognition, pp 4854–4863

  31. Gao J, Zhang T, Xu C (2019) Graph convolutional tracking. In: IEEE conference on computer vision and pattern recognition, pp 4644–4654

  32. Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  33. Li B, Wu W, Zhu Z, Yan J, Hu X (2020) High performance visual tracking with siamese region proposal network. In: IEEE conference on computer vision and pattern recognition, pp 6668–6677

  34. Zhang Z, Peng H (2019) Deeper and wider Siamese networks for real-time visual tracking. In: IEEE conference on computer vision and pattern recognition, pp 4586–4595

  35. Chen Z, Zhong B, Li G, Zhang S, Ji R (2020) Siamese box adaptive network for visual tracking. In: IEEE conference on computer vision and pattern recognition, pp 6667–6676

  36. Guo D, Wang J, Cui Y, Wang Z, Chen S (2020) SiamCAR: Siamese fully convolutional classification and regression for visual tracking. In: IEEE conference on computer vision and pattern recognition, pp 6268–6276

  37. Yang K, He Z et al (2021) SiamCorners: Siamese corner networks for visual tracking. IEEE Trans Multimed. https://doi.org/10.1109/TMM.2021.3074239

  38. Cui Y, Jiang C, Wang L, Wu G (2020) Fully convolutional online tracking. In: arXiv Preprint arXiv:2004.07109

  39. Zhang Z, Peng H, Fu J, Li B, Hu W (2020) Ocean: Object-aware anchor-free tracking. In: European conference on computer vision, pp 771–787

  40. Uzkent B, Rangnekar A, Hoffman M (2019) Tracking in aerial hyperspectral videos using deep kernelized correlation filters. IEEE Trans Geosci Remote Sens 57(1):449–461

    Article  Google Scholar 

  41. Chen L, Zhao Y (2021) Mosaic spatial-spectral feature based object tracking in hyperspectral video. In: IEEE workshop on hyperspectral image and signal processing: evolution in remote sensing

  42. Zhang Z, Qian K, Du J, Zhou H (2021) Multi-features integration based hyperspectral videos tracker. In: IEEE workshop on hyperspectral image and signal processing: evolution in remote sensing

  43. Ronneberger O, Fischer P, Brox T (2015) U-Net: Convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention, pp 234–241

  44. Woo S, Park J, Lee JY, Kweon I (2018) CBAM: Convolutional block attention module. In: European conference on computer vision, pp 3–19

  45. Cipolla R, Y, G, Kendall A (2018) Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: IEEE conference on computer vision and pattern recognition, pp 7482–7491

  46. Danelljan M, Bhat G, Khan F, Felsberg M (2017) ECO: Efficient convolution operators for tracking. In: IEEE conference on computer vision and pattern recognition, pp 6931–6939

  47. Valmadre J, Bertinetto L, Henriques J, Vedaldi A, Torr P (2017) End-to-End representation learning for correlation filter based tracking. In: IEEE conference on computer vision and pattern recognition, pp 5000–5008

  48. Choi J, Chang H et al. (2018) Context-aware deep feature compression for high-speed visual tracking. In: IEEE conference on computer vision and pattern recognition, pp 479–488

  49. Danelljan M, Bhat G et al. (2019) ATOM: Accurate tracking by overlap maximization. In: IEEE conference on computer vision and pattern recognition, pp 4655–4664

  50. Wang Q, Zhang L, Bertinetto L, Hu W, Torr P (2019) Fast online object tracking and segmentation: A unifying approach. In: IEEE conference on computer vision and pattern recognition, pp 1328–1338

  51. Li X, Ma C et al. (2019) Target-aware deep tracking. In: IEEE conference on computer vision and pattern recognition, pp 1369–1378

  52. Lukezic A, Matas J, Kristan M (2020) D3S-A discriminative single shot segmentation tracker. In: IEEE conference on computer vision and pattern recognition, pp 7131–7140

  53. Guo D, Shao Y et al. (2021) Graph attention tracking. In: IEEE conference on computer vision and pattern recognition

  54. Li Z, Xiong F, Zhou J, Wang J, Lu J, Qian Y (2020) BAENet: A band attention aware ensemble network for hyperspectral object tracking. In: IEEE international conference on image processing, pp 2106–2110

Download references

Acknowledgements

This work was funded by the National Natural Science Foundation of China under Grant 42071302 and the Innovation Program for Chongqing Overseas Returnees under Grant cx2019144. Thanks to NVIDIA Corporation for the support of GPU device. The authors are grateful to Fengchao Xiong, who is with the School of Computer Science and engineering of Nanjing university of science and technology, for providing the HOT dataset.

Author information

Authors and Affiliations

  1. Key Laboratory of Optoelectronic Technology and Systems of the Education Ministry of China, Chongqing University, Chongqing, 400044, China

    Yiming Tang, Hong Huang, Yufei Liu & Yuan Li

  2. State Key Laboratory of Coal Mine Disaster Dynamics and Control, Chongqing University, Chongqing, 400044, China

    Hong Huang

  3. Collaborative Innovation Center for Brain Science, Chongqing University, Chongqing, 400044, China

    Yufei Liu

  4. Centre for NanoHealth, College of Engineering, Swansea University, Singleton Park, Swansea, SA2 8PP, UK

    Yufei Liu

Authors
  1. Yiming Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hong Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yufei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Hong Huang or Yufei Liu.

Ethics declarations

Conflicts of interest

All authors declare that they have no conflicts of interest that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tang, Y., Huang, H., Liu, Y. et al. A Siamese network-based tracking framework for hyperspectral video. Neural Comput & Applic 35, 2381–2397 (2023). https://doi.org/10.1007/s00521-022-07712-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07712-5

Keywords

Search

Navigation