Skip to main content
SpringerLink
Log in

Learning sequence-to-sequence affinity metric for near-online multi-object tracking

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

In this paper, we propose a sequence-to-sequence affinity metric for the data association of near-online multi-object tracking. The proposed metric learns the affinity between track sequence consisting of the already associated detections and hypothesis sequence consisting of detections in the near future. With the potential hypothesis sequences, we leverage the idea that if a track sequence has a high affinity for a hypothesis sequence, and the hypothesis sequence also shares a close affinity for a current detection, then the affinity between the track sequence and the detection is high. By using the short hypothesis sequence as a “bridge”, the proposed sequence-to-sequence affinity metric enhances the conventional track sequence to detection affinity metric and improves its robustness to object occlusion and missing. Besides, in order to eliminate the negative effects of false alarms, we propose a false alarm model using both appearance and scale features of detection. The robustness of the proposed affinity metric allows us to use a simple greedy data association algorithm. Experimental results on the challenging MOT16 and MOT17 benchmarks demonstrate the effectiveness of our method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. In: 12th \(\{\)USENIX\(\}\) symposium on operating systems design and implementation (\(\{\)OSDI\(\}\) 16), pp 265–283

  2. Bernardin K, Stiefelhagen R (2008) Evaluating multiple object tracking performance: the clear mot metrics. J Image Video Process. https://doi.org/10.1155/2008/246309

    Article  Google Scholar 

  3. Bochinski E, Eiselein V, Sikora T (2017) High-speed tracking-by-detection without using image information. In: IEEE international conference on advanced video and signal based surveillance (AVSS). IEEE, pp 1–6

  4. Breitenstein MD, Reichlin F, Leibe B, Koller-Meier E, Van Gool L (2011) Online multiperson tracking-by-detection from a single, uncalibrated camera. IEEE Trans Pattern Anal Mach Intell 33(9):1820–1833

    Article  Google Scholar 

  5. Chen J, Sheng H, Zhang Y, Xiong Z (2017) Enhancing detection model for multiple hypothesis tracking. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. IEEE, pp 18–27

  6. Choi W (2015) Near-online multi-target tracking with aggregated local flow descriptor. In: Proceedings of the IEEE international conference on computer vision. IEEE, pp 3029–3037

  7. Chopra S, Hadsell R, LeCun Y (2005) Learning a similarity metric discriminatively, with application to face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 539–546

  8. Chu Q, Ouyang W, Li H, Wang X, Liu B, Yu N (2017) Online multi-object tracking using CNN-based single object tracker with spatial–temporal attention mechanism. In: Proceedings of the IEEE international conference on computer vision. IEEE, pp 4836–4845

  9. Dehghan A, Modiri Assari S, Shah M (2015) Gmmcp tracker: Globally optimal generalized maximum multi clique problem for multiple object tracking. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 4091–4099

  10. Fagot-Bouquet L, Audigier R, Dhome Y, Lerasle F (2016) Improving multi-frame data association with sparse representations for robust near-online multi-object tracking. In: Proceedings of the European conference on computer vision. Springer, pp 774–790

  11. Feng W, Hu Z, Wu W, Yan J, Ouyang W (2019) Multi-object tracking with multiple cues and switcher-aware classification. arXiv:1901.06129

  12. Fu Z, Feng P, Angelini F, Chambers J, Naqvi SM (2018) Particle phd filter based multiple human tracking using online group-structured dictionary learning. IEEE Access 6:14764–14778

    Article  Google Scholar 

  13. Henschel R, Leal-Taixe L, Cremers D, Rosenhahn B (2018) Fusion of head and full-body detectors for multi-object tracking. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. IEEE, pp 1428–1437

  14. Keuper M, Tang S, Andres B, Brox T, Schiele B (2018) Motion segmentation and multiple object tracking by correlation co-clustering. IEEE Trans Pattern Anal Mach Intell 42(1):140–153

    Article  Google Scholar 

  15. Kim C, Li F, Ciptadi A, Rehg JM (2015) Multiple hypothesis tracking revisited. In: Proceedings of the IEEE international conference on computer vision. IEEE, pp 4696–4704

  16. Kim C, Li F, Rehg JM (2018) Multi-object tracking with neural gating using bilinear LSTM. In: Proceedings of the European conference on computer vision. Springer, pp 200–215

  17. Kuhn HW (1955) The hungarian method for the assignment problem. Nav Res Logist Q 2(1–2):83–97

    Article  MathSciNet  Google Scholar 

  18. Kutschbach T, Bochinski E, Eiselein V, Sikora T (2017) Sequential sensor fusion combining probability hypothesis density and kernelized correlation filters for multi-object tracking in video data. In: IEEE international conference on advanced video and signal based surveillance (AVSS). IEEE, pp 1–5

  19. Lan L, Tao D, Gong C, Guan N, Luo Z (2016) Online multi-object tracking by quadratic pseudo-boolean optimization. In: International joint conference on artificial intelligence, pp 3396–3402

  20. Lan L, Wang X, Zhang S, Tao D, Gao W, Huang TS (2018) Interacting tracklets for multi-object tracking. IEEE Trans Image Process 27(9):4585–4597

    Article  MathSciNet  Google Scholar 

  21. Lan L, Wang X, Hua G, Huang TS, Tao D (2020) Semi-online multi-people tracking by re-identification. Int J Comput Vis 128:1937–1955

    Article  MathSciNet  Google Scholar 

  22. Leal-Taixé L, Canton-Ferrer C, Schindler K (2016) Learning by tracking: siamese CNN for robust target association. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. IEEE, pp 33–40

  23. Liao S, Hu Y, Zhu X, Li SZ (2015) Person re-identification by local maximal occurrence representation and metric learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 2197–2206

  24. Long C, Haizhou A, Zijie Z, Chong S (2018) Real-time multiple people tracking with deeply learned candidate selection and person re-identification. In: IEEE international conference on multimedia and expo (ICME). IEEE, pp 1–6

  25. Ma C, Yang C, Yang F, Zhuang Y, Zhang Z, Jia H, Xie X (2018) Trajectory factory: Tracklet cleaving and re-connection by deep siamese bi-gru for multiple object tracking. In: IEEE international conference on multimedia and expo (ICME). IEEE, pp 1–6

  26. Milan A, Rezatofighi SH, Dick A, Reid I, Schindler K (2017) Online multi-target tracking using recurrent neural networks. In: Proceedings of the AAAI conference on artificial intelligence, AAAI

  27. Pirsiavash H, Ramanan D, Fowlkes CC (2011) Globally-optimal greedy algorithms for tracking a variable number of objects. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 1201–1208

  28. Redmon J, Farhadi A (2018) Yolov3: an incremental improvement. arXiv:1804.02767

  29. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

  30. Sadeghian A, Alahi A, Savarese S (2017) Tracking the untrackable: learning to track multiple cues with long-term dependencies. In: Proceedings of the IEEE international conference on computer vision. IEEE, pp 300–311

  31. Sanchez-Matilla R, Poiesi F, Cavallaro A (2016) Online multi-target tracking with strong and weak detections. In: Proceedings of the European conference on computer vision. Springer, pp 84–99

  32. Son J, Baek M, Cho M, Han B (2017) Multi-object tracking with quadruplet convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 5620–5629

  33. Wang B, Wang L, Shuai B, Zuo Z, Liu T, Luk Chan K, Wang G (2016) Joint learning of convolutional neural networks and temporally constrained metrics for tracklet association. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. IEEE, pp 1–8

  34. Wang J, Song Y, Leung T, Rosenberg C, Wang J, Philbin J, Chen B, Wu Y (2014) Learning fine-grained image similarity with deep ranking. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 1386–1393

  35. Xiang Y, Alahi A, Savarese S (2015) Learning to track: online multi-object tracking by decision making. In: Proceedings of the IEEE international conference on computer vision. IEEE, pp 4705–4713

  36. Yang B, Nevatia R (2014) Multi-target tracking by online learning a CRF model of appearance and motion patterns. Int J Comput Vis 107(2):203–217

    Article  MathSciNet  Google Scholar 

  37. Yoon YC, Boragule A, Song YM, Yoon K, Jeon M (2018) Online multi-object tracking with historical appearance matching and scene adaptive detection filtering. In: IEEE international conference on advanced video and signal based surveillance (AVSS). IEEE, pp 1–6

  38. Zhang L, Li Y, Nevatia R (2008) Global data association for multi-object tracking using network flows. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 1–8

  39. Zhu J, Yang H, Liu N, Kim M, Zhang W, Yang MH (2018) Online multi-object tracking with dual matching attention networks. In: Proceedings of the European conference on computer vision. Springer, pp 366–382

Download references

Acknowledgements

This work is supported by China Scholarship Council (CSC) and National Natural Science Foundation of China (NSFC No. 61906210, Researches on Detection based Visual Multi-object Tracking).

Author information

Authors and Affiliations

  1. National University of Defense Technology, Changsha, China

    Weijiang Feng, Long Lan, Xiang Zhang & Zhigang Luo

Authors
  1. Weijiang Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Long Lan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhigang Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weijiang Feng.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feng, W., Lan, L., Zhang, X. et al. Learning sequence-to-sequence affinity metric for near-online multi-object tracking. Knowl Inf Syst 62, 3911–3930 (2020). https://doi.org/10.1007/s10115-020-01488-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-020-01488-7

Keywords

Search

Navigation