Skip to main content
SpringerLink
Log in

Learning behavior patterns from video for agent-based crowd modeling and simulation

  • Published:
Autonomous Agents and Multi-Agent Systems Aims and scope Submit manuscript

Abstract

This paper proposes a novel data-driven modeling framework to construct agent-based crowd model based on real-world video data. The constructed crowd model can generate crowd behaviors that match those observed in the video and can be used to predict trajectories of pedestrians in the same scenario. In the proposed framework, a dual-layer architecture is proposed to model crowd behaviors. The bottom layer models the microscopic collision avoidance behaviors, while the top layer models the macroscopic crowd behaviors such as the goal selection patterns and the path navigation patterns. An automatic learning algorithm is proposed to learn behavior patterns from video data. The learned behavior patterns are then integrated into the dual-layer architecture to generate realistic crowd behaviors. To validate its effectiveness, the proposed framework is applied to two different real world scenarios. The simulation results demonstrate that the proposed framework can generate crowd behaviors similar to those observed in the videos in terms of crowd density distribution. In addition, the proposed framework can also offer promising performance on predicting the trajectories of pedestrians.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. The source code, training data and simulation videos can be downloaded from: http://crowds.sce.ntu.edu.sg/resource.html.

  2. The original video and extracted trajectories are downloaded from http://www.ee.cuhk.edu.hk/~xgwang/grandcentral.html.

  3. The video and extracted trajectories are downloaded from http://www.vision.ee.ethz.ch/datasets/index.en.html.

References

  1. Ali, S., Nishino, K., Manocha, D., & Shah, M. (2013). Modeling, Simulation and Visual Analysis of Crowds. New York: Springer.

    Book  Google Scholar 

  2. Ali, S., & Shah, M. (2007). A lagrangian particle dynamics approach for crowd flow segmentation and stability analysis. In IEEE Conference on Computer Vision and Pattern Recognition, 2007 (CVPR’07) (pp. 1–6). Piscataway: IEEE.

  3. Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In Computer Vision-ECCV 2004 (pp. 25–36). Berlin: Springer.

  4. Finnsson, H., & Björnsson, Y. (2008). Simulation-based approach to general game playing. In Proceedings of the 23rd national conference on Artificial intelligence (Vol. 1, pp. 259–264).

  5. Ge, W., Collins, R., & Ruback, R. (2012). Vision-based analysis of small groups in pedestrian crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(5), 1003–1016.

    Article  Google Scholar 

  6. Gratch, J., & Marsella, S. (2004). A domain-independent framework for modeling emotion. Cognitive Systems Research, 5(4), 269–306.

    Article  Google Scholar 

  7. Guy, S.J., Lin, M.C., & Manocha, D. (2010). Modeling collision avoidance behavior for virtual humans. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems(AAMAS’10) (pp. 575–582).

  8. Guy, S. J., Van Den Berg, J., Liu, W., Lau, R., Lin, M. C., & Manocha, D. (2012). A statistical similarity measure for aggregate crowd dynamics. ACM Transactions on Graphics (TOG), 31(6), 190.

    Article  Google Scholar 

  9. Helbing, D., Johansson, A., & Al-Abideen, H. Z. (2007). Dynamics of crowd disasters: An empirical study. Physical Review E, 75(4), 046109.

    Article  Google Scholar 

  10. Helbing, D., & Molnar, P. (1995). Social force model for pedestrian dynamics. Physical Review E, 51(5), 4282.

    Article  Google Scholar 

  11. Jin, X., Xu, J., Wang, C. C., Huang, S., & Zhang, J. (2008). Interactive control of large-crowd navigation in virtual environments using vector fields. IEEE Computer Graphics and Applications, 6, 37–46.

    Google Scholar 

  12. Johansson, A., Helbing, D., & Shukla, P. K. (2007). Specification of the social force pedestrian model by evolutionary adjustment to video tracking data. Advances in Complex Systems, 10(supp02), 271–288.

    Article  MathSciNet  MATH  Google Scholar 

  13. Ju, E., Choi, M. G., Park, M., Lee, J., Lee, K. H., & Takahashi, S. (2010). Morphable crowds. ACM Transactions on Graphics, 29, 140.

    Article  Google Scholar 

  14. Kaminka, G. A., & Fridman, N. (2006). A cognitive model of crowd behavior based on social comparison theory. In Proceedings of the AAAI-2006 workshop on cognitive modeling.

  15. Lee, K.H., Choi, M.G., Hong, Q., & Lee, J. (2007). Group behavior from video: A data-driven approach to crowd simulation. In Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Computer animation (pp. 109–118). Eurographics Association.

  16. Lerner, A., Chrysanthou, Y., & Lischinski, D. (2007). Crowds by example. Computer Graphics Forum, 26, 655–664.

    Article  Google Scholar 

  17. Lerner, A., Fitusi, E., Chrysanthou, Y., & Cohen-Or, D. (2009). Fitting behaviors to pedestrian simulations. In Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (pp. 199–208). New York: ACM.

  18. Lin, D., Grimson, E., & Fisher, J. (2009). Learning visual flows: A lie algebraic approach. In IEEE Conference on Computer Vision and Pattern Recognition, 2009 (CVPR’09) (pp. 747–754). Piscataway: IEEE.

  19. Lin, D., Grimson, E., & Fisher, J. (2010). Modeling and estimating persistent motion with geometric flows. In IEEE Conference on Computer Vision and Pattern Recognition, 2010 (CVPR’10 ) (pp. 1–8). Piscataway: IEEE.

  20. Luo, L., Zhou, S., Cai, W., Low, M. Y. H., Tian, F., Wang, Y., et al. (2008). Agent-based human behavior modeling for crowd simulation. Computer Animation and Virtual Worlds, 19(3–4), 271–281.

    Article  Google Scholar 

  21. Mehran, R., Moore, B.E., & Shah, M. (2010). A streakline representation of flow in crowded scenes. In European conference on computer vision, 2010 (ECCV’10) (pp. 439–452). Berlin: Springer.

  22. Musse, S. R., Jung, C. R., Jacques, J., & Braun, A. (2007). Using computer vision to simulate the motion of virtual agents. Computer Animation and Virtual Worlds, 18(2), 83–93.

    Article  Google Scholar 

  23. Musse, S. R., & Thalmann, D. (2001). Hierarchical model for real time simulation of virtual human crowds. IEEE Transactions on Visualization and Computer Graphics, 7(2), 152–164.

    Article  Google Scholar 

  24. Pan, X., Han, C. S., Dauber, K., & Law, K. H. (2007). A multi-agent based framework for the simulation of human and social behaviors during emergency evacuations. AI & Society, 22(2), 113–132.

    Article  Google Scholar 

  25. Pellegrini, S., Ess, A., Schindler, K., & Van Gool, L. (2009). You’ll never walk alone: Modeling social behavior for multi-target tracking. In IEEE International Conference on Computer Vision, 2009 (ICCV’09) (pp. 261–268). Piscataway: IEEE.

  26. Schmidt, M., & Lipson, H. (2009). Distilling free-form natural laws from experimental data. Science, 324(5923), 81–85.

    Article  Google Scholar 

  27. Scovanner, P., & Tappen, M.F. (2009). Learning pedestrian dynamics from the real world. In IEEE International Conference on Computer Vision, 2009 (ICCV’09) (Vol. 9, pp. 381–388).

  28. Shao, W., & Terzopoulos, D. (2005). Autonomous pedestrians. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation (pp. 19–28).

  29. Thalmann, D. (2007). Crowd Simulation. New York: Wiley Online Library.

    Book  Google Scholar 

  30. Tomasi, C., & Kanade, T. (1991). Detection and Tracking of Point Features. Pittsburgh: School of Computer Science, Carnegie Mellon University.

    Google Scholar 

  31. Tsai, J., Fridman, N., Bowring, E., Brown, M., Epstein, S., Kaminka, G., Marsella, S., Ogden, A., Rika, I., Sheel, A., et al. (2011). Escapes: Evacuation simulation with children, authorities, parents, emotions, and social comparison. In Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’11) (pp. 457–464).

  32. Van den Berg, J., Lin, M., & Manocha, D. (2008). Reciprocal velocity obstacles for real-time multi-agent navigation. In IEEE International Conference on Robotics and Automation, 2008 (ICRA’08) (pp. 1928–1935). Piscataway: IEEE.

  33. Welford, B. (1962). Note on a method for calculating corrected sums of squares and products. Technometrics, 4(3), 419–420.

    Article  MathSciNet  Google Scholar 

  34. Wolinski, D., Guy, S. J., Olivier, A.-H., Lin, M., Manocha, D., & Pettré, J. (2014). Parameter estimation and comparative evaluation of crowd simulations. Computer Graphics Forum, 33, 303–312.

    Article  Google Scholar 

  35. Yamaguchi, K., Berg, A. C., Ortiz, L. E., & Berg, T. L. (2011). Who are you with and where are you going? In IEEE Conference on Computer Vision and Pattern Recognition, 2011 (CVPR’11) (pp. 1345–1352). Piscataway: IEEE.

  36. Zhao, M., Turner, S. J., & Cai, W. (2013). A data-driven crowd simulation model based on clustering and classification. In 2013 IEEE/ACM 17th International Symposium on Distributed Simulation and Real Time Applications (DS-RT) (pp. 125–134). Piscataway: IEEE.

  37. Zhong, J., & Cai, W. (2015). Differential evolution with sensitivity analysis and the powell’s method for crowd model calibration. Journal of Computational Science, 9, 26–32.

    Article  Google Scholar 

  38. Zhong, J., Cai, W., Luo, L., & Lees, M. (2014). Ea-based evacuation planning using agent-based crowd simulation. In Proceedings of the 2014 Winter Simulation Conference (pp. 395–406). Piscataway: IEEE Press.

  39. Zhong, J., Cai, W., Luo, L., & Yin, H. (2015). Learning behavior patterns from video: A data-driven framework for agent-based crowd modeling. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS’15) (pp. 801–809). International Foundation for Autonomous Agents and Multiagent Systems.

  40. Zhong, J., Hu, N., Cai, W., Lees, M., & Luo, L. (2015). Density-based evolutionary framework for crowd model calibration. Journal of Computational Science, 6, 11–22.

    Article  Google Scholar 

  41. Zhong, J., Luo, L., Cai, W., & Lees, M. (2014). Automatic rule identification for agent-based crowd models through gene expression programming. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems (AAMAS’14) (pp. 1125–1132). International Foundation for Autonomous Agents and Multiagent Systems.

  42. Zhong, J., Ong, Y.-S., & Cai, W. (2016). Self-learning gene expression programming. IEEE Transactions on Evolutionary Computation, 20(1), 65–80.

    Article  Google Scholar 

  43. Zhou, B., Wang, X., & Tang, X. (2012). Understanding collective crowd behaviors: Learning a mixture model of dynamic pedestrian-agents. In IEEE Conference on Computer Vision and Pattern Recognition, 2012 CVPR’12 (pp. 2871–2878). Piscataway: IEEE.

  44. Zhou, S., Chen, D., Cai, W., Luo, L., Low, M. Y. H., Tian, F., et al. (2010). Crowd modeling and simulation technologies. ACM Transactions on Modeling and Computer Simulation, 20(4), 20.

    Article  Google Scholar 

Download references

Acknowledgments

The research reported in this paper is financially supported by the Tier 1 Academic Research Fund (AcRF) under Project Number RG23/14. Linbo Luo is supported by National Natural Science Foundation of China (Grant No. 61502370), China 111 Project (No. B16037) and Fundamental Research Funds for the Central Universities (Grant No. JB150305).

Author information

Authors and Affiliations

  1. School of Computer Engineering, Nanyang Technological University, Singapore, Singapore

    Jinghui Zhong, Wentong Cai & Mingbi Zhao

  2. School of Cyber Engineering, Xidian University, Xi’an, China

    Linbo Luo

Authors
  1. Jinghui Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wentong Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linbo Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mingbi Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinghui Zhong.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhong, J., Cai, W., Luo, L. et al. Learning behavior patterns from video for agent-based crowd modeling and simulation. Auton Agent Multi-Agent Syst 30, 990–1019 (2016). https://doi.org/10.1007/s10458-016-9334-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10458-016-9334-8

Keywords

Search

Navigation