Skip to main content
SpringerLink
Log in

IST-PTEPN: an improved pedestrian trajectory and endpoint prediction network based on spatio-temporal information

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

The prediction of pedestrian trajectories in complicated dynamic situations has garnered a great deal of interest among researchers and academics, and it plays a crucial role in numerous domains, including autonomous vehicles, intelligent robotics, and video surveillance. In this study, we offer the IST-PTEPN, a trainable and interpretable end-to-end model for predicting pedestrian trajectory. IST-PTEPN encodes the spatial and temporal characteristics of pedestrian trajectories and surrounding scenes with CNN and Transformer, then feeds the encoded vectors into Endpoint Classify CNN to generate predicted endpoints of the trajectories, and finally combines TCN and GAN to generate high-quality pedestrian trajectories. Experiments on two public datasets, ETH and UCY, demonstrate that our IST-PTEPN pedestrian trajectory prediction and endpoint prediction method outperforms the mainstream state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

All data generated or analyzed during this study are included in this published article.

References

  1. Movassagh AA, Alzubi JA, Gheisari M, Rahimi M, Mohan S, Abbasi AA, Nabipour N (2021) Artificial neural networks training algorithm integrating invasive weed optimization with differential evolutionary model. J Ambient Intell Humanized Comput 14:1–9

    Google Scholar 

  2. Chen J, Li K, Li K, Yu PS, Zeng Z (2021) Dynamic planning of bicycle stations in dockless public bicycle-sharing system using gated graph neural network. ACM Transact Intell Syst Technol (TIST) 12(2):1–22

    Article  Google Scholar 

  3. Ferrer G, Sanfeliu A (2014) Bayesian human motion intentionality prediction in urban environments. Pattern Recogn Lett 44:134–140

    Article  Google Scholar 

  4. Wang C, Zhu H, Hao Q, Xiao K, Xiong H (2021) Variable interval time sequence modeling for career trajectory prediction: deep collaborative perspective. In: Proceedings of the Web Conference 2021, pp. 612–623

  5. Yang C, Sun M, Zhao WX, Liu Z, Chang EY (2017) A neural network approach to jointly modeling social networks and mobile trajectories. ACM Transact Inform Syst (TOIS) 35(4):1–28

    Article  Google Scholar 

  6. Santos R, Pardo XM, Fdez-Vidal XR (2019) Scene wireframes sketching for unmanned aerial vehicles. Pattern Recogn 86:354–367

    Article  Google Scholar 

  7. Luo Y, Cai P, Bera A, Hsu D, Lee WS, Manocha D (2018) Porca: Modeling and planning for autonomous driving among many pedestrians. IEEE Robot Automat Lett 3(4):3418–3425

    Article  Google Scholar 

  8. Morotomi K, Katoh M, Hayashi H, Toyota Motor Corp (2014) Collision position predicting device. U.S. Patent 8,849,558

  9. Raksincharoensak P, Hasegawa T, Nagai M (2016) Motion planning and control of autonomous driving intelligence system based on risk potential optimization framework. Int J Automot Eng 7(AVEC14):53–60

    Article  Google Scholar 

  10. Luber M, Stork JA, Tipaldi GD, Arras KO (2010) People tracking with human motion predictions from social forces. In: 2010 IEEE international conference on robotics and automation, pp. 464–469. IEEE.

  11. Chen J, Li K, Li K, Yu PS, Zeng Z (2021) Dynamic bicycle dispatching of dockless public bicycle-sharing systems using multi-objective reinforcement learning. ACM Transact Cyber-Phys Syst (TCPS) 5(4):1–24

    Article  Google Scholar 

  12. Alzubi JA, Jain R, Nagrath P, Satapathy S, Taneja S, Gupta P (2021) Deep image captioning using an ensemble of CNN and LSTM based deep neural networks. J Intell Fuzzy Syst 40(4):5761–5769

    Article  Google Scholar 

  13. Giuliari F, Hasan I, Cristani M, Galasso F (2021) Transformer networks for trajectory forecasting. In: 2020 25th international conference on pattern recognition (ICPR) (pp. 10335–10342). IEEE

  14. Chen J, Li K, Rong H, Bilal K, Li K, Philip SY (2019) A periodicity-based parallel time series prediction algorithm in cloud computing environments. Inf Sci 496:506–537

    Article  Google Scholar 

  15. Kitani, K.M., Ziebart, B.D., Bagnell, J.A. and Hebert, M., 2012. Activity forecasting. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7–13, 2012, Proceedings, Part IV 12 (pp. 201–214). Springer Berlin Heidelberg.

  16. Deo N, Trivedi MM (2017) Learning and predicting on-road pedestrian behavior around vehicles. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC) (pp. 1–6). IEEE

  17. Rehder E, Kloeden H (2015) Goal-directed pedestrian prediction. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (pp. 50–58)

  18. Kim K, Lee D, Essa I (2011) Gaussian process regression flow for analysis of motion trajectories. In: 2011 International Conference on Computer Vision, pp. 1164–1171. IEEE.

  19. Xu YW, Cao XB, Li T (2009) Extended Kalman filter based pedestrian localization for collision avoidance. In: 2009 International Conference on Mechatronics and Automation (pp. 4366–4370). IEEE

  20. Alahi A, Goel K, Ramanathan V, Robicquet A, Fei-Fei L, Savarese S (2016) Social lstm: human trajectory prediction in crowded spaces. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 961–971)

  21. Li Y (2019) Which way are you going? Imitative decision learning for path forecasting in dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 294–303)

  22. Liang J, Jiang L, Niebles JC, Hauptmann AG, Fei-Fei L (2019) Peeking into the future: predicting future person activities and locations in videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5725–5734)

  23. Yu C, Ma X, Ren J, Zhao H, Yi S (2020) Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16 (pp. 507–523). Springer International Publishing

  24. Mohamed A, Qian K, Elhoseiny M, Claudel C (2020) Social-stgcnn: a social spatio-temporal graph convolutional neural network for human trajectory prediction. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14424–14432)

  25. Pellegrini S, Ess A, Schindler K, Van Gool L (2009) You'll never walk alone: modeling social behavior for multi-target tracking. In: 2009 IEEE 12th international conference on computer vision (pp. 261–268). IEEE

  26. Lerner A, Chrysanthou Y, Lischinski D (2007) Crowds by example. In: Computer graphics forum (vol. 26, No. 3, pp. 655–664). Blackwell Publishing Ltd, Oxford

  27. Helbing D, Molnar P (1995) Social force model for pedestrian dynamics. Phys Rev E 51(5):4282

    Article  Google Scholar 

  28. Schöller C, Aravantinos V, Lay F, Knoll A (2020) What the constant velocity model can teach us about pedestrian motion prediction. IEEE Robot Automat Lett 5(2):1696–1703

    Article  Google Scholar 

  29. Vemula A, Muelling K, Oh J (2018) Social attention: modeling attention in human crowds. In 2018 IEEE international Conference on Robotics and Automation (ICRA) (pp. 4601–4607). IEEE

  30. Zhang P, Ouyang W, Zhang P, Xue J, Zheng N (2019) Sr-lstm: state refinement for lstm towards pedestrian trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12085–12094)

  31. Sadeghian A, Kosaraju V, Sadeghian A, Hirose N, Rezatofighi H, Savarese S (2019) Sophie: an attentive gan for predicting paths compliant to social and physical constraints. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1349–1358)

  32. Zamboni S, Kefato ZT, Girdzijauskas S, Norén C, Dal Col L (2022) Pedestrian trajectory prediction with convolutional neural networks. Pattern Recogn 121:108252

    Article  Google Scholar 

  33. Huang Y, Bi H, Li Z, Mao T, Wang Z (2019) Stgat: modeling spatial-temporal interactions for human trajectory prediction. In: Proceedings of the IEEE/CVF international conference on computer vision (pp. 6272–6281)

  34. Kosaraju V, Sadeghian A, Martín-Martín R, Reid I, Rezatofighi H, Savarese S (2019) Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. Adv Neural Inform Process Syst 32:1–10

    Google Scholar 

  35. Sun J, Jiang Q, Lu C (2020) Recursive social behavior graph for trajectory prediction. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 660–669)

  36. Zhao Z, Liu C (2021) STUGCN: a social spatio-temporal unifying graph convolutional network for trajectory prediction. In 2021 6th International Conference on Automation, Control and Robotics Engineering (CACRE) (pp. 546–550). IEEE

  37. Gupta A, Johnson J, Fei-Fei L, Savarese S, Alahi A (2018) Social gan: socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2255–2264)

  38. Li J, Ma H, Tomizuka M (2019) Conditional generative neural system for probabilistic trajectory prediction. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 6150–6156). IEEE

  39. Eiffert S, Li K, Shan M, Worrall S, Sukkarieh S, Nebot E (2020) Probabilistic crowd GAN: Multimodal pedestrian trajectory prediction using a graph vehicle-pedestrian attention network. IEEE Robot Automat Lett 5(4):5026–5033

    Article  Google Scholar 

  40. Li Y, Liang R, Wei W, Wang W, Zhou J, Li X (2021) Temporal pyramid network with spatial-temporal attention for pedestrian trajectory prediction. IEEE Transact Network Sci Eng 9(3):1006–1019

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by the National Natural Science Foundation of China (61573182, 62073164), and by the Fundamental Research Funds for the Central Universities (NS2022041).

Author information

Authors and Affiliations

  1. College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, China

    Xin Yang, Jiangfeng Fan & Siyuan Xing

Authors
  1. Xin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiangfeng Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Siyuan Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Yang.

Ethics declarations

Conflict of interest

Xin Yang declares that he has no conflict of interest. Jiangfeng Fan declares that he has no conflict of interest. Siyuan Xing declares that he has no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, X., Fan, J. & Xing, S. IST-PTEPN: an improved pedestrian trajectory and endpoint prediction network based on spatio-temporal information. Int. J. Mach. Learn. & Cyber. 14, 4193–4206 (2023). https://doi.org/10.1007/s13042-023-01889-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-023-01889-4

Keywords

Search

Navigation