Skip to main content
SpringerLink
Log in

An online multiple object tracker based on structure keeper net

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

We propose a novel online multiple object tracker taking structure information into account. State-of-the-art multi-object tracking (MOT) approaches commonly focus on discriminative appearance features, while neglect in different levels structure information and the core of data association. Addressing this, we design a new tracker fully exploiting structure information and encoding such information into the cost function of the graph matching model. Firstly, a new measurement is proposed to compare the structure similarity of two graphs whose nodes are equal. With this measurement, we define a complete matching which performs association in high efficiency. Secondly, for incomplete matching scenarios, a structure keeper net (SKnet) is designed to adaptively establish the graph for matching. Finally, we conduct extensive experiments on benchmarks including MOT2015 and MOT17. The results demonstrate the competitiveness and practicability of our tracker.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. More details about solution could be found in the FGM [55].

  2. Since the global matrix is 30∗30 which is too large to show in the paper, we give the corresponding quantitative results of the examples in the Supplementary Materials.

  3. We also compare the robustness of a few state-of-the-art online trackers with our tracker on MOT17 training sets. For saving space, the results and analyses are shown in Supplementary Materials.

  4. In tracking, GT means the real trajectories of the objects in the video sequence.

  5. Since our local evaluating toolkit does not contain it ,we only show it in the testing comparisons.

References

  1. Touil DE, Terki N, Medouakh S (2018) Learning spatially correlation filters based on convolutional features via PSO algorithm and two combined color spaces for visual tracking. Appl Intell 48:2837–2846

    Article  Google Scholar 

  2. Huang W, Gu J, Ma X, Li Y (2020) End-to-end multitask Siamese network with residual hierarchical attention for real-time object tracking. Appl Intell 50:1908–1921

    Article  Google Scholar 

  3. Kumar A, Walia GS, Sharma K (2020) A novel approach for multi-cue feature fusion for robust object tracking. Appl Intell 50:3201–3218

    Article  Google Scholar 

  4. Li T, Wu P, Ding F, et al. (2020) Parallel dual networks for visual object tracking applied intelligence. https://doi.org/10.1007/s10489-020-01783-4

  5. Parate MR, Satpute VR, Bhurchandi KM (2018) Global-patch-hybrid template-based arbitrary object tracking with integral channel features. Appl Intell 48:300–314

    Article  Google Scholar 

  6. Cao TP, Elton D, Deng G (2012) Fast buffering for FPGA implementation of vision-based object recognition systems. J Real-Time Image Proc 7(3):173–183

    Article  Google Scholar 

  7. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR 2014, pp 580–587

  8. Girshick R (2015) Fast r-cnn. In: ICCV 2015, pp 1440–1448

  9. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR 2016, pp 770–778

  10. Kaushal M, Khehra BS (2017) BBBCO and fuzzy entropy based modified background subtraction algorithm for object detection in videos. Appl Intell 47:1008–1021

    Article  Google Scholar 

  11. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: NIPS 2015, pp 91–99

  12. Bae SH, Yoon KJ (2018) Confidence-based data association and discriminative deep appearance learning for robust online multi-object tracking. PAMI 40(3):595–610

    Article  Google Scholar 

  13. Yang M, Jia Y (2016) Temporal dynamic appearance modeling for online multi-person tracking. Comput Vis Image Underst 153:16–28

    Article  Google Scholar 

  14. Bergmann P, Meinhardt T, Leal-Taixé L (2019) Tracking without bells and whistles. In: ICCV 2019, pp 941–951

  15. Novak CL, Shafer SA (1992) Anatomy of a color histogram. In: CVPR 1992, pp 599–605

  16. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: CVPR 2005, pp 886–893

  17. Choi W (2015) Near-online multi-target tracking with aggregated local flow descriptor. In: ICCV 2015, pp 3029–3037

  18. Henschel R, Zou Y, Rosenhahn B (2019) Multiple people tracking using body and joint detections. In: CVPR 2019 Workshops

  19. Henschel R, Leal-Taixé L, Cremers D, Rosenhahn B (2018) Fusion of head and full-body detectors for multi-object tracking. In: CVPR 2018 Workshops, pp 1428–1437

  20. Xu Y, Osep A, Ban Y, Horaud R, Leal-Taixé L, Alamedapineda X (2020) How To Train Your Deep Multi-Object Tracker. In: CVPR 2020, pp 6787–6796

  21. Andriyenko A, Schindler K (2011) Multi-target tracking by continuous energy minimization. In: CVPR 2011, 2(6), pp 7

  22. Choi W, Savarese S (2010) Multiple target tracking in world coordinate with single, minimally calibrated camera. In: ECCV 2010, pp 553–567

  23. Hong Yoon J, Lee CR, Yang MH, Yoon KJ (2016) Online multi-object tracking via structural constraint event aggregation. In: CVPR 2016, pp 1392–1400

  24. Bochinski E, Eiselein V, Sikora T (2017) High-Speed tracking-by-detection without using image information. In: AVSS 2017, pp 1–6

  25. Leal-Taixé L, Pons-Moll G, Rosenhahn B (2011) Everybody needs somebody: Modeling social and grouping behavior on a linear programming multiple people tracker. In: ICCV 2011 Workshops, pp 120–127

  26. Pellegrini S, Ess A, Schindler K, Van Gool L (2009) You’ll never walk alone: Modeling social behavior for multi-target tracking. In: ICCV 2009, pp 261–268

  27. Scovanner P, Tappen MF (2009) Learning pedestrian dynamics from the real world. In: ICCV 2009, pp 381–388

  28. Yamaguchi K, Berg AC, Ortiz LE, Berg TL (2011) Who are you with and where are you going?. In: CVPR 2011, pp 1345–1352

  29. Fang K, Xiang Y, Li X, Savarese S (2018) Recurrent autoregressive networks for online multi-object tracking. In: WACV 2018, pp 466–475

  30. Kim C, Li F, Rehg JM (2018) Multi-object tracking with neural gating using bilinear lstm. In: ECCV 2018, pp 200–215

  31. Milan A, Rezatofighi SH, Dick AR, Reid ID, Schindler K (2017) Online multi-target tracking using recurrent neural networks. In: AAAI 2017, pp 4225–4232

  32. Sadeghian A, Alahi A, Savarese S (2017) Tracking the untrackable: Learning to track multiple cues with long-term dependencies. In: ICCV 2017, pp 300–311

  33. Yoon YC, Boragule A, Song YM, Yoon K, Jeon M (2018) Online multi-object tracking with historical appearance matching and scene adaptive detection filtering. In: AVSS 2018, pp 1–6

  34. Chu Q, Ouyang W, Li H, Wang X, Liu B, Yu N (2017) Online multi-object tracking using CNN-based single object tracker with spatial-temporal attention mechanism. In: ICCV 2017, pp 4846–4855

  35. Leal-Taixé L, Canton-Ferrer C, Schindler K. (2016) Learning by tracking: Siamese CNN for robust target association. In: CVPR 2016 Workshops, pp 33–40

  36. Wang N, Zou Q, Ma Q, et al. (2020) A light tracker for online multiple pedestrian tracking. J Real-Time Image Proc. https://doi.org/10.1007/s11554-020-00962-3

  37. Tang S, Andriluka M, Andres B, Schiele B (2017) Multiple people tracking by lifted multicut and person re-identification. In: CVPR 2017, pp 3539–3548

  38. Brasó G, Leal-Taixe L (2020) Learning a neural solver for multiple object tracking. In: CVPR 2020, pp 6247–6257

  39. Leibe B, Schindler K, Van Gool L (2007) Coupled detection and trajectory estimation for multi-object tracking. In: ICCV 2007, pp 1–8

  40. Keuper M, Tang S, Zhongjie Y, Andres B, Brox T, Schiele B (2016) A multi-cut formulation for joint segmentation and tracking of multiple objects. arXiv:1607.06317

  41. Kim S, Kwak S, Feyereisl J, Han B (2012) Online multi-target tracking by large margin structured learning. In: ACCV 2012, pp 98–111

  42. Bae SH, Yoon KJ (2014) Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning. In: CVPR2014, pp 1218–1225

  43. Wang G, Wang Y, Zhang H, Gu R, Hwang JN (2018) Exploit the connectivity: Multi-object tracking with trackletnet. arXiv:1811.07258

  44. Sheng H, Zhang Y, Chen J, Xiong Z, Zhang J (2018) Heterogeneous association graph fusion for target association in multiple object tracking. IEEE Trans Circuits Syst Video Technol 29(11):3269–3280

    Article  Google Scholar 

  45. Yang M, Wu Y, Jia Y (2017) A hybrid data association framework for robust online multi-object tracking. IEEE Trans Image Process 26(12):5667–5679

    Article  MathSciNet  Google Scholar 

  46. Kim C, Li F, Ciptadi A, Rehg JM (2015) Multiple hypothesis tracking revisited. In: ICCV 2015, pp 4696–4704

  47. Sheng H, Chen J, Zhang Y, Ke W, Xiong Z, Yu J (2018) Iterative multiple hypothesis tracking with tracklet-level association. IEEE Trans Circuits Syst Video Technol 29(12):3660– 3672

    Article  Google Scholar 

  48. Son J, Baek M, Cho M, Han B (2017) Multi-object tracking with quadruplet convolutional neural networks. In: CVPR 2017, pp 5620–5629

  49. Wang B, Wang L, Shuai B, Zuo Z, Liu T, Luk Chan K, Wang G (2016) Joint learning of convolutional neural networks and temporally constrained metrics for tracklet association. In: CVPR 2016 Workshops, pp 1–8

  50. Shen X, Sui X, Pan K, Tao Y (2016) Adaptive pedestrian tracking via patch-based features and spatial–temporal similarity measurement. Pattern Recogn 53:163–173

    Article  Google Scholar 

  51. Battaglia PW, Hamrick JB, Bapst V, et al. (2018) Relational inductive biases, deep learning, and graph networks. arXiv:1806.01261

  52. Li Y, Gu C, Dullien T, Vinyals O, Kohli P (2019) Graph matching networks for learning the similarity of graph structured objects. arXiv:1904.12787

  53. Yu HX, Wu A, Zheng WS (2020) Unsupervised person re-identification by deep asymmetric metric embedding. PAMI 42(4):956–973

    Article  Google Scholar 

  54. Li J, Liu H (2017) Projective low-rank subspace clustering via learning deep encoder. In: IJCAI 2017, pp 2145–2151

  55. Zhou F, De la Torre F (2012) Factorized graph matching. In: CVPR 2012, pp 127–134

  56. Leal-Taixé L, Milan A, Reid I, Roth S, Schindler K (2015) Motchallenge 2015: Towards a benchmark for multi-target tracking. arXiv:1504.01942

  57. Milan A, Leal-Taixé L, Reid I, Roth S, Schindler K (2016) MOT16: A benchmark for multi-object tracking. arXiv:1603.00831

  58. Dollár P, Appel R, Belongie S, Perona P (2014) Fast feature pyramids for object detection. PAMI 36(8):1532–1545

    Article  Google Scholar 

  59. Felzenszwalb P, McAllester D, Ramanan D (2008) A discriminatively trained, multiscale, deformable part model. In: CVPR 2008, pp 1–8

  60. Yang F, Choi W, Lin Y (2016) Exploit all the layers: Fast and accurate cnn object detector with scale dependent pooling and cascaded rejection classifiers. In: CVPR 2016, pp 2129–2137

  61. Vedaldi A, Lenc K (2015) Matconvnet: Convolutional neural networks for matlab. In: Proceedings of the 23rd ACM international conference on Multimedia 2015, pp 689–692

  62. Bernardin K, Stiefelhagen R (2008) Evaluating multiple object tracking performance: The CLEAR MOT metrics. J Image Video Process 2008(1):1–10

    Google Scholar 

  63. Ristani E, Solera F, Zou R, Cucchiara R, Tomasi C (2016) Performance measures and a data set for multi-target, multi-camera tracking. In: ECCV workshop on benchmarking multi-target tracking

  64. Munkres J (1957) Algorithms for the assignment and transportation problems. J Soc Ind Appl Math 5(1):32–38

    Article  MathSciNet  Google Scholar 

  65. Xu J, Cao Y, Zhang Z, Hu H (2019) Spatial-temporal relation networks for multi-object tracking. In: ICCV 2019, pp 3988–3998

  66. Xiang J, Xu G, Ma C, Hou J (2020) End-to-end learning deep crf models for multi-object tracking. IEEE Transactions on Circuits and Systems for Video Technology

  67. Chen L, Ai H, Shang C, Zhuang Z, Bai B (2017) Online multi-object tracking with convolutional neural networks. In: ICIP 2017, pp 645–649

  68. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: CVPR 2009, pp 248–255

  69. Zheng L, Shen L, Tian L, Wang S, Wang J, Tian Q (2015) Scalable person re-identification: A benchmark. In: ICCV 2015, pp 1116–1124

  70. Li W, Zhao R, Xiao T, Wang X (2014) Deepreid: Deep filter pairing neural network for person re-identification. In: CVPR 2014, pp 152–159

  71. Sanchez-Matilla R, Poiesi F, Cavallaro A (2016) Online multi-target tracking with strong and weak detections. In: ECCV 2016, pp 84–99

  72. Liu Q, Liu B, Wu Y, Li W, Yu N (2019) Real-time online multi-object tracking in compressed domain. IEEE Access 7:76489–76499

    Article  Google Scholar 

  73. Chu P, Ling H (2019) FAMNet: joint learning of feature. In: Affinity and multi-dimensional assignment for online multiple object tracking. arXiv:1904.04989

  74. Chu P, Fan H, Tan CC, Ling H (2019) Online multi-object tracking with instance-aware tracker and dynamic model refreshment. In: WACV 2019, pp 161–170

  75. Sheng H, Hao L, Chen J, Zhang Y, Ke W (2017) Robust local effective matching model for multi-target tracking. In: Pacific Rim conference on multimedia, pp 233–243

  76. Wojke N, Paulus D (2016) Global data association for the probability hypothesis density filter using network flows. In: ICRA 2016, pp 567–572

  77. Mahgoub H, Mostafa K, Wassif KT, Farag I (2017) Multi-target tracking using hierarchical convolutional features and motion cues. International Journal of Advanced Computer Science and Applications, 8(11):217–222

  78. Manen S, Timofte R, Dai D, Van Gool L (2016) Leveraging single for multi-target tracking using a novel trajectory overlap affinity measure. In: WACV 2016, pp 1–9

  79. Wu H, Hu Y, Wang K, Li H, Nie L, Cheng H (2019) Instance-aware representation learning and association for online multi-person tracking. Pattern Recogn 94:25–34

    Article  Google Scholar 

  80. Zhou H, Ouyang W, Cheng J, Wang X, Li H (2018) Deep continuous conditional random fields with asymmetric inter-object constraints for online multi-object tracking. IEEE Trans Circ Syst Video Technol 29(4):1011–1022

    Article  Google Scholar 

  81. Xiang Y, Alahi A, Savarese S (2015) Learning to track: online multi-object tracking by decision making. In: ICCV 2015, pp 4705–4713

  82. Zhou X, Jiang P, Wei Z, Dong H, Wang F (2018) Online multi-object tracking with structural invariance constraint. In: BMVC

  83. Kieritz H, Becker S, Hübner W, Arens M (2016) Online multi-person tracking using integral channel features. In: AVSS 2016, pp 122–130

  84. Maksai A, Fua P (2019) Eliminating exposure bias and metric mismatch in multiple object tracking. In: CVPR 2019, pp 4639–4648

  85. Chen J, Sheng H, Zhang Y, Xiong Z Enhancing detection model for multiple hypothesis tracking. In: CVPR 2017 Workshops, pp 18–27

  86. Yoon K, Gwak J, Song YM, Yoon YC, Jeon MG (2020) OneShotDA: Online multi-object tracker with one-shot-learning-based data association. IEEE Access 8:38060–38072

    Article  Google Scholar 

  87. Fu Z, Angelini F, Chambers J, Naqvi SM (2019) Multi-level cooperative fusion of GM-PHD filters for online multiple human tracking. IEEE Trans Multimed 21(9):2277–2291

    Article  Google Scholar 

  88. Chu Q, Ouyang W, Liu B, Zhu F, Yu N (2020) DASOT: a unified framework integrating data association and single object tracking for online multi-object tracking. Proc AAAI Confer Artif Intell 34(7):10672–10679

    Google Scholar 

  89. Zhu J, Yang H, Liu N, Kim M, Zhang W, Yang MH (2018) Online multi-object tracking with dual matching attention networks. In: ECCV 2018, pp 366–382

  90. Lee SH, Kim MY, Bae SH (2018) Learning discriminative appearance models for online multi-object tracking with appearance discriminability measures. IEEE Access 6:67316–67328

    Article  Google Scholar 

  91. Fu Z, Feng P, Angelini F, Chambers J, Naqvi SM (2018) Particle phd filter based multiple human tracking using online group-structured dictionary learning. IEEE Access 6:14764– 14778

    Article  Google Scholar 

  92. Peng J, Wang T, Lin W, Wang J, See J, Wen S, Ding E (2020) TPM: multiple object tracking with tracklet-plane matching pattern recognition. https://doi.org/10.1016/j.patcog.2020.107480

Download references

Author information

Authors and Affiliations

  1. Beijing Key Laboratory of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China

    Nan Wang, Qi Zou, Qiulin Ma, Yaping Huang & Xiaoyu Wu

  2. College of Automation Engineering, Shanghai University of Electric Power, Shanghai, China

    Haitao Lou

  3. School of Information Management, Beijing Information Science and Technology University, Beijing, China

    Huiyong Liu

Authors
  1. Nan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiulin Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yaping Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haitao Lou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaoyu Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Huiyong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qi Zou.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

(PDF 4.29 MB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, N., Zou, Q., Ma, Q. et al. An online multiple object tracker based on structure keeper net. Appl Intell 51, 8010–8029 (2021). https://doi.org/10.1007/s10489-021-02294-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02294-6

Keywords

Search

Navigation