Abstract
Incorporating multiple cameras is an effective solution to improve the performance and robustness of multi-target tracking to occlusion and appearance ambiguities. In this paper, we propose a new multi-camera multi-target tracking method based on a space-time-view hyper-graph that encodes higher-order constraints (i.e., beyond pairwise relations) on 3D geometry, appearance, motion continuity, and trajectory smoothness among 2D tracklets within and across different camera views. We solve tracking in each single view and reconstruction of tracked trajectories in 3D environment simultaneously by formulating the problem as an efficient search of dense sub-hypergraphs on the space-time-view hyper-graph using a sampling based approach. Experimental results on the PETS 2009 dataset and MOTChallenge 2015 3D benchmark demonstrate that our method performs favorably against the state-of-the-art methods in both single-camera and multi-camera multi-target tracking, while achieving close to real-time running efficiency. We also provide experimental analysis of the influence of various aspects of our method to the final tracking performance.









Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Many methods do not form tracklets but perform association directly on detections in each frame. In this work, we unify these methods by treating individual frame detections as tracklets of length one.
The last frame index of \({\mathcal {T}}\) and the first frame index of \({\mathcal {T}}'\) may correspond to multiple detections from different camera views.
Note that this is different from the degree of the nodes, which specifies how many hyper-edges can associate with one node.
The \(\beta \)-subhypergraph indicates the sub-hypergraph of STV hyper-graph, which includes \(\beta \) nodes.
The calculation of the number of hyper-edges, including nodes \(\nu \), \(\nu '\) and \(\nu _j\) is a combinational problem, that is to choose \(k-3\) nodes from the reliable node set \(\varOmega _i-\{\nu , \nu ', \nu _j\}\). Specifically, we set \(\rho _i = 0\) for \(|\varOmega _i| < 3\), since there does not exist enough nodes to construct a hyper-edge in that case.
We will make our method and our implementation of Hofmann et al. (2013) along with the tracking results available after the paper decision.
Since different input detections and ground truth are used, it is unfair to directly compare the tracking results of the proposed method with the results presented in Hofmann et al. (2013).
References
Andriyenko, A., & Schindler, K. (2011). Multi-target tracking by continuous energy minimization. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1265–1272).
Andriyenko, A., Schindler, K., & Roth, S. (2012). Discrete-continuous optimization for multi-target tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1926–1933).
Attanasi, A., Cavagna, A., Castello, L. D., Giardina, I., Jelic, A., Melillo, S., et al. (2015). GReTA—a novel global and recursive tracking algorithm in three dimensions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(1), 1.
Berclaz, J., Fleuret, F., & Fua, P. (2009). Multiple object tracking using flow linear programming. In Winter-PETS (pp. 1–8). Snowbird: IEEE.
Berclaz, J., Fleuret, F., Türetken, E., & Fua, P. (2011). Multiple object tracking using k-shortest paths optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(9), 1806–1819.
Breitenstein, M. D., Reichlin, F., Leibe, B., Koller-Meier, E., & Gool, L. J. V. (2011). Online multi-person tracking-by-detection from a single, uncalibrated camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(9), 1820–1833.
Brendel, W., Amer, M. R., & Todorovic, S. (2011). Multiobject tracking as maximum weight independent set. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1273–1280).
Chari, V., Lacoste-Julien, S., Laptev, I., & Sivic, J. (2015). On pairwise costs for network flow multi-object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 5537–5545).
Dehghan, A., Tian, Y., Torr, P. H. S., & Shah, M. (2015). Target identity-aware network flow for online multiple target tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1146–1154).
Dollár, P., Wojek, C., Schiele, B., & Perona, P. (2012). Pedestrian detection: An evaluation of the state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(4), 743–761.
Felzenszwalb, P. F., McAllester, D. A., & Ramanan, D. (2008). A discriminatively trained, multiscale, deformable part model. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).
Ferryman, J. M., & Shahrokni, A. (2009). PETS2009: Dataset and challenge. In Winter-PETS (pp. 1–6).
Fleuret, F., Berclaz, J., Lengagne, R., & Fua, P. (2008). Multicamera people tracking with a probabilistic occupancy map. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2), 267–282.
Hofmann, M., Wolf, D., & Rigoll, G. (2013). Hypergraphs for joint multi-view reconstruction and multi-object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 3650–3657).
Hong, L., & Cui, N. (2000). An interacting multipattern joint probabilistic data association (imp-jpda) algorithm for multitarget tracking. Signal Processing, 80(8), 1561–1575.
Huang, C., Li, Y., & Nevatia, R. (2013). Multiple target tracking by learning-based hierarchical association of detection responses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(4), 898–910.
Isard, M., & Blake, A. (1998). Condensation—conditional density propagation for visual tracking. International Journal of Computer Vision, 29(1), 5–28.
Izadinia, H., Saleemi, I., Li, W., & Shah, M. (2012) (MP)\(^2\)T: Multiple people multiple parts tracker. In Proceedings of European Conference on Computer Vision (pp. 100–114).
Jiang, H., Fels, S., & Little, J. J. (2007). A linear programming approach for multiple object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).
Khan, Z., Balch, T. R., & Dellaert, F. (2005). MCMC-based particle filtering for tracking a variable number of interacting targets. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(11), 1805–1918.
Kim, J., Dai, Y., Li, H., Du, X., & Kim, J. (2013). Multi-view 3D reconstruction from uncalibrated radially-symmetric cameras. In Proceedings of IEEE International Conference on Computer Vision (pp. 1896–1903).
Klinger, T., Rottensteiner, F., & Heipke, C. (2015). Probabilistic multi-person tracking using dynamic bayes networks. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, II–3/W5, 435–442.
Kostrikov, I., Horbert, E., & Leibe, B. (2014). Probabilistic labeling cost for high-accuracy multi-view reconstruction. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1534–1541).
Kuhn, W., & Tucker, A. (1951) Nonlinear programming. In Proceedings of 2nd Berkeley Symposium (pp. 481–492).
Kuo, C. H., & Nevatia, R. (2011). How does person identity recognition help multi-person tracking? In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1217–1224).
Leal-Taixé, L., Milan, A., Reid, I.D., Roth, S., & Schindler, K. (2015). Motchallenge 2015: towards a benchmark for multi-target tracking. CoRR abs/1504.01942.
Leal-Taixé, L., Pons-Moll, G., & Rosenhahn, B. (2011). Everybody needs somebody: modeling social and grouping behavior on a linear programming multiple people tracker. In Workshops in Conjunction with IEEE International Conference on Computer Vision (pp. 120–127).
Leal-Taixé, L., Pons-Moll, G., & Rosenhahn, B. (2012) Branch-and-price global optimization for multi-view multi-object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1987–1994).
Leven, W. F., & Lanterman, A. D. (2009). Unscented kalman filters for multiple target tracking with symmetric measurement equations. IEEE Transaction on Automatic Control, 54(2), 370–375.
Liu, H., & Yan, S. (2012). Efficient structure detection via random consensus graph. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 574–581).
Liu, H., Yang, X., Latecki, L. J., & Yan, S. (2012). Dense neighborhoods on affinity graph. IJCV, 98(1), 65–82.
Liu, Y., Li, H., & Chen, Y. Q. (2012). Automatic tracking of a large number of moving targets in 3d. In Proceedings of European Conference on Computer Vision (pp. 730–742).
Marchesotti, L., Marcenaro, L., Ferrari, G., & Regazzoni, C. S. (2002) Multiple object tracking under heavy occlusions by using kalman filters based on shape matching. In Proceedings of IEEE International Conference on Image Processing (pp. 341–344).
Milan, A. (2011) Continuous energy minimization tracker. http://www.milanton.de/contracking/index.html.
Milan, A., Leal-Taixé, L., Schindler, K., Roth, S., & Reid, I.D. (2015). Multiple object tracking benchmark: 3d mot. https://motchallenge.net/results/3D_MOT_2015/.
Milan, A., Roth, S., & Schindler, K. (2014). Continuous energy minimization for multitarget tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(1), 58–72.
Ojala, T., Pietikäinen, M., & Mäenpää, T. (2000). Gray scale and rotation invariant texture classification with local binary patterns. In Proceedings of European Conference on Computer Vision (pp. 404–420).
Pellegrini, S., Ess, A., Schindler, K., & Gool, L. J. V. (2009). You’ll never walk alone: modeling social behavior for multi-target tracking. In Proceedings of IEEE International Conference on Computer Vision (pp. 261–268).
Pirsiavash, H., Ramanan, D., & Fowlkes, C. C. (2011). Globally-optimal greedy algorithms for tracking a variable number of objects. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1201–1208).
Reid, D. B. (1979). An algorithm for tracking multiple targets. IEEE Transactions on Automatic Control, 24, 843–854.
Shi, X., Ling, H., Hu, W., Yuan, C., & Xing, J. (2014). Multi-target tracking with motion context in tensor power iteration. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 3518–3525).
Shu, G., Dehghan, A., Oreifej, O., Hand, E., & Shah, M. (2012). Part-based multiple-person tracking with partial occlusion handling. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1815–1821).
Smith, K., Gatica-Perez, D., & Odobez, J. M. (2005). Using particles to track varying numbers of interacting people. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 962–969).
Stiefelhagen, R., Bernardin, K., Bowers, R., Garofolo, J. S., Mostefa, D., & Soundararajan, P. (2006). The CLEAR 2006 evaluation. CLEAR (pp. 1–44). Berlin: Springer.
Wen, L., Li, W., Yan, J., Lei, Z., Yi, D., & Li, S. Z. (2014) Multiple target tracking based on undirected hierarchical relation hypergraph. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (pp. 3457–3464).
Wu, Z., Hristov, N.I., Kunz, T. H., & Betke, M. (2009). Tracking-reconstruction or reconstruction-tracking? Comparison of two multiple hypothesis tracking approaches to interpret 3D object motion from several camera views. In Proceedings of the IEEE Workshop on Motion and Video Computing (pp. 1–8).
Wu, Z., Kunz, T. H., & Betke, M. (2011). Efficient track linking methods for track graphs using network-flow and set-cover techniques. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1185–1192).
Yang, B., & Nevatia, R. (2012). Multi-target tracking by online learning of non-linear motion patterns and robust appearance models. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1918–1925).
Yang, B., & Nevatia, R. (2012). An online learned CRF model for multi-target tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 2034–2041).
Yang, M., Liu, Y., Wen, L., You, Z., & Li, S. Z. (2014). A probabilistic framework for multitarget tracking with mutual occlusions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).
Yu, Q., & Medioni, G. G. (2009). Multiple-target tracking by spatiotemporal monte carlo markov chain data association. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(12), 2196–2210.
Zhang, L., Li, Y., & Nevatia, R. (2008). Global data association for multi-object tracking using network flows. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).
Zhou, D., Huang, J., & Schölkopf, B. (2006). Learning with hypergraphs: Clustering, classification, and embedding. Advances in Neural Information Processing Systems (pp. 1601–1608). Cambridge: MIT Press.
Acknowledgments
We would like to thank Dawei Du for a number of suggestions that considerably improved the quality of this paper. Longyin Wen and Siwei Lyu were supported by US National Science Foundation Research Grant (CCF-1319800). Zhen Lei was supported by the National Key Research and Development Plan (Grant No. 2016 YFC0801002), the Chinese National Natural Science Foundation Projects #61375037, #61473291. Honggang Qi was supported by National Nature Science Foundation of China #61472388.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Hiroshi Ishikawa, Takeshi Masuda, Yasuyo Kita and Katsushi Ikeuchi.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (avi 11348 KB)
Rights and permissions
About this article
Cite this article
Wen, L., Lei, Z., Chang, MC. et al. Multi-Camera Multi-Target Tracking with Space-Time-View Hyper-graph. Int J Comput Vis 122, 313–333 (2017). https://doi.org/10.1007/s11263-016-0943-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-016-0943-0