Skip to main content
SpringerLink
Log in

Structural Constraint Data Association for Online Multi-object Tracking

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Online two-dimensional (2D) multi-object tracking (MOT) is a challenging task when the objects of interest have similar appearances. In that case, the motion of objects is another helpful cue for tracking and discriminating multiple objects. However, when using a single moving camera for online 2D MOT, observable motion cues are contaminated by global camera movements and, thus, are not always predictable. To deal with unexpected camera motion, we propose a new data association method that effectively exploits structural constraints in the presence of large camera motion. In addition, to reduce incorrect associations with mis-detections and false positives, we develop a novel event aggregation method to integrate assignment costs computed by structural constraints. We also utilize structural constraints to track missing objects when they are re-detected again. By doing this, identities of the missing objects can be retained continuously. Experimental results validated the effectiveness of the proposed data association algorithm under unexpected camera motions. In addition, tracking results on a large number of benchmark datasets demonstrated that the proposed MOT algorithm performs robustly and favorably against various online methods in terms of several quantitative metrics, and that its performance is comparable to offline methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. \(\eta \) is set to 0.5 in our experiments.

  2. iris.usc.edu/people/yangbo/downloads.html.

  3. cvlibs.net/datasets/kitti/eval_tracking.php.

  4. motchallenge.net/data/2D_MOT_2015/.

References

  • Bae, S. H., Yoon, K. J. (2014). Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning. InProceedings of IEEE conference on computer vision and pattern recognition (pp. 1218–1225).

  • Bar-Shalom, Y., & Li, X. R. (1995). Multitarget-multisensor tracking: Principles and techniques. Storrs, CT: YBS Publishing.

    Google Scholar 

  • Bernardin, K., & Stiefelhagen, R. (2008). Evaluating multiple object tracking performance: The clear mot metrics. EURASIP Journal on Image and Video Processing, 1(1–1), 10.

    Google Scholar 

  • Betke, M., & Wu, Z. (2016). Data Association for multi-object visual tracking. Synthesis lectures on computer vision. Morgan & Claypool. https://books.google.co.kr/books?id=tn0cvgAACAAJ.

  • Blackman, S., & Popoli, R. (1999). Design and analysis of modern tracking systems. Norwood: Artech House Radar Library, Artech House.

    MATH  Google Scholar 

  • Breitenstein, M. D., Reichlin, F., Leibe, B., Koller-Meier, E., & Van Gool, L. (2011). Online multiperson tracking-by-detection from a single, uncalibrated camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(9), 1820–1833.

    Article  Google Scholar 

  • Choi, W. (2015). Near-online multi-target tracking with aggregated local flow descriptor. In Proceedings of IEEE international conference on computer vision (pp. 3029–3037).

  • Dollar, P., Appel, R., Belongie, S., & Perona, P. (2014). Fast feature pyramids for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(8), 1532–1545.

    Article  Google Scholar 

  • Duan, G., Ai, H., Cao, S., & Lao, S. (2012). Group tracking: Exploring mutual relations for multiple object tracking. In Proceedings of European conference on computer vision (pp. 129–143).

  • Ess, A., Leibe, B., Schindler, K., & Gool, L. V. (2008). A mobile vision system for robust multi-person tracking. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Everingham, M., Gool, L., Williams, C. K., Winn, J., & Zisserman, A. (2010). The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2), 303–338.

    Article  Google Scholar 

  • Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9), 1627–1645.

    Article  Google Scholar 

  • Geiger, A. (2013). Probabilistic models for 3D urban scene understanding from movable platforms. Ph.D. thesis, KIT.

  • Geiger, A., Lauer, M., Wojek, C., Stiller, C., & Urtasun, R. (2014). 3D traffic scene understanding from movable platforms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(5), 1012–1025.

    Article  Google Scholar 

  • Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013). Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11), 1231–1237.

    Article  Google Scholar 

  • Grabner, H., Matas, J., Gool, L. J. V., & Cattin, P. C. (2010). Tracking the invisible: Learning where the object might be. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1285–1292).

  • Kim, C., Li, F., Ciptadi, A., & Rehg, J. M. (2015). Multiple hypothesis tracking revisited. In Proceedings of IEEE international conference on computer vision (pp. 4696–4704).

  • Kim, S., Kwak, S., Feyereisl, J., & Han, B. (2012). Online multi-target tracking by large margin structured learning. In Proceedings of Asian conference on computer vision (pp. 98–111).

  • Kuhn, H. W. (1955). The hungarian method for the assignment problem. Naval Research Logistics, 2(1–2), 83–97.

    Article  MathSciNet  MATH  Google Scholar 

  • Leal-Taixe, L., Canton-Ferrer, C., & Schindler, K. (2016). Learning by tracking: Siamese CNN for robust target association. In Proceedings of IEEE conference on computer vision and pattern recognition workshops (pp. 33–40).

  • Leal-Taixé, L., Fenzi, M., Kuznetsova, A., Rosenhahn, B., & Savarese, S. (2014). Learning an image-based motion context for multiple people tracking. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 3542–3549).

  • Leal-Taixé, L., Pons-Moll, G., & Rosenhahn, B. (2011). Everybody needs somebody: Modeling social and grouping behavior on a linear programming multiple people tracker. In Proceedings of IEEE international conference on computer vision workshop

  • Lenz, P., Geiger, A., & Urtasun, R. (2015). Followme: Efficient online min-cost flow tracking with bounded memory and computation. In Proceedings of IEEE international conference on computer vision (pp. 4364–4372).

  • Li, X. R., & Jilkov, V. P. (2003). Survey of maneuvering target tracking. Part i. Dynamic models. IEEE Transactions on Aerospace and Electronic Systems, 39(4), 1333–1364.

    Article  Google Scholar 

  • Li, Y., Huang, C., & Nevatia, R. (2009). Learning to associate: Hybridboosted multi-target tracker for crowded scene. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 2953–2960).

  • McLaughlin, N., del Rincón, J. M., & Miller, P. C. (2015). Enhancing linear programming with motion modeling for multi-target tracking. In IEEE winter conference on applications of computer vision (pp. 71–77).

  • Milan, A., Leal-Taixé, L., Schindler, K., & Reid, I. (2015). Joint tracking and segmentation of multiple targets. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Milan, A., Roth, S., & Schindler, K. (2014). Continuous energy minimization for multitarget tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(1), 58–72.

    Article  Google Scholar 

  • Milan, A., Schindler, K., & Roth, S. (2013). Detection- and trajectory-level exclusion in multiple object tracking. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Pellegrini, S., Ess, A., Schindler, K., & Gool, L. V. (2009). You’ll never walk alone: Modeling social behavior for multi-target tracking. In Proceedings of IEEE international conference on computer vision (pp. 261–268).

  • Pirsiavash, H., Ramanan, D., & Fowlkes, C. C. (2011). Globally-optimal greedy algorithms for tracking a variable number of objects. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1201–1208).

  • Poiesi, F., Mazzon, R., & Cavallaro, A. (2013). Multi-target tracking on confidence maps: An applicaion to people tracking. In CVIU (pp. 257–1272).

  • Rezatofighi, H., Milan, A., Zhang, Z., Shi, Q., Dick, A., & Reid, I. (2015). Joint probabilistic data association revisited. In Proceedings of IEEE international conference on computer vision (pp. 3047–3055).

  • Rezatofighi, S. H., Milan, A., Zhang, Z., Shi, Q., Dick, A., & Reid, I. (2016). Joint probabilistic matching using m-Best solutions. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 136–145).

  • Takala, V., Pietikäinen, M. (2007). Multi-object tracking using color, texture and motion. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Wang, S., & Fowlkes, C. (2015). Learning optimal parameters for multi-target tracking. In Proceedings of the British machine vision conference (pp. 4.1–4.13).

  • Wang, X., Yang, M., Zhu, S., & Lin, Y. (2015). Regionlets for generic object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(10), 2071–2084.

    Article  Google Scholar 

  • Wu, Z., & Betke, M. (2016). Global optimization for coupled detection and data association in multiple object tracking. Computer Vision and Image Understanding, 143, 25–37.

    Article  Google Scholar 

  • Xiang, Y., Alahi, A., & Savarese, S. (2015). Learning to track: Online multi-object tracking by decision making. In Proceedings of IEEE international conference on computer vision (pp. 4705–4713).

  • Xing, J., Ai, H., & Lao, S. (2009). Multi-object tracking through occlusions by local tracklets filtering and global tracklets association with detection responses. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1200–1207).

  • Yang, B., & Nevatia, R. (2012). Multi-target tracking by online learning of non-linear motion patterns and robust appearance models. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1918–1925).

  • Yang, B., & Nevatia, R. (2014). Multi-target tracking by online learning a CRF model of appearance and motion patterns. International Journal of Computer Vision, 107(2), 203–217.

    Article  MathSciNet  MATH  Google Scholar 

  • Yoon, J. H., Lee, C. R., Yang, M. H., & Yoon, K. J. (2016). Online multi-object tracking via structural constraint event aggregation. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1392–1400).

  • Yoon, J. H., Yang, M. H., Lim, J., & Yoon, K. J. (2015). Bayesian multi-object tracking using motion context from multiple objects. In Proceedings of IEEE winter conference on applications of computer vision (pp. 33–40).

  • Zhang, L., & van der Maaten, L. (2013). Structure preserving object tracking. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1838–1845).

Download references

Acknowledgements

This work was partially supported by ‘The Cross-Ministry Giga KOREA Project’ grant funded by the Korea government (MSIT) (No. GK18P0200, Development of 4D reconstruction and dynamic deformable action model based hyper-realistic service technology and No. GK18P0300, Real-time 4D reconstruction of dynamic objects for ultra-realistic service). The work was also partially supported by IITP grant funded by the Korea government (MSIP) (2014-0-00059) and Samsung Research Funding Center of Samsung Electronics under Project Number SRFC-TC1603-05. M.-H. Yang is supported in part by the the NSF CAREER Grant #1149783.

Author information

Authors and Affiliations

  1. Korea Electronics Technology Institute, Seongnam, Gyeonggi-do, Republic of Korea

    Ju Hong Yoon

  2. Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology, Gwangju, Republic of Korea

    Chang-Ryeol Lee

  3. Electrical Engineering and Computer Science, University of California at Merced, Merced, CA, 95344, USA

    Ming-Hsuan Yang

  4. Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology, KAIST, 291 Daehak-ro, Yuseong-gu, Daejeon, 34141, Republic of Korea

    Kuk-Jin Yoon

Authors
  1. Ju Hong Yoon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chang-Ryeol Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ming-Hsuan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kuk-Jin Yoon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kuk-Jin Yoon.

Additional information

Communicated by Robert T. Collins.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yoon, J.H., Lee, CR., Yang, MH. et al. Structural Constraint Data Association for Online Multi-object Tracking. Int J Comput Vis 127, 1–21 (2019). https://doi.org/10.1007/s11263-018-1087-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-018-1087-1

Keywords

Search

Navigation