Skip to main content
SpringerLink
Log in

PTP-STGCN: Pedestrian Trajectory Prediction Based on a Spatio-temporal Graph Convolutional Neural Network

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

It is the prerequisite to ensure the safety of road users in traffic scenes for the application of autonomous vehicles. Pedestrians are the main participants in traffic scenes, and reasonable inference and prediction of their future trajectories are crucial for autonomous driving technology and road safety. Pedestrian trajectories are highly dynamic, apparently random, and complex to interact with traffic environment agents; therefore, selective modeling of spatial interaction and temporal dependence of pedestrians is necessary. To address this challenge, this paper proposes a novel pedestrian trajectory prediction model based on a spatio-temporal graph convolutional network (PTP-STGCN). Specifically, a new crowd interaction attention (CIA) function is defined to quantify the interaction information between adjacent pedestrians better. This function captures the spatial interaction features of pedestrians at each time step by a spatial graph convolution network (S-GCN). Meanwhile, the time-series motion features of each pedestrian are extracted by a temporal transformer network (T-transformer), and a spatio-temporal interaction graph of pedestrian features is constructed by the STGCN composed of the S-GCN and T-transformer. Finally, a time-extrapolator convolutional neural network (TXP-CNN) is used to predict pedestrian trajectories in the time dimension of the STGCN. The experimental results on the ETH and UCY datasets show that the proposed model achieves better performance than the state-of-the-art baselines regarding the average displacement error (ADE) and final displacement error (FDE). Due to the excellent performance in pedestrian trajectory prediction, the proposed network achieves more predictive final planned trajectory of an autonomous vehicle, while avoiding unnecessary trajectory changes and collision risk, thus providing a promising solution for practical pedestrian trajectory prediction applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Berenguer A D, Alioscha-Perez M, Oveneke M C, Sahli H (2021) Context-aware human trajectories prediction via latent variational model. IEEE Trans Circ Syst Video Technol 31(5):1876–1889

    Article  Google Scholar 

  2. Ma Q, Zou Q, Huang Y, Wang N (2021) Dynamic pedestrian trajectory forecasting with LSTM-based Delaunay triangulations. Applied Intelligence

  3. Ji Y, Yang Y, Shen F, Shen H T, Li X (2020) A survey of human action analysis in hri applications. IEEE Trans Circ Syst Video Technol 30(7):2114–2128

    Article  Google Scholar 

  4. Wang Z, Liu S, Zhang J, Chen S, Guan Q (2016) A spatio-temporal crf for human interaction understanding. IEEE Trans Circ Syst Video Technol 27(8):1647–1660

    Article  Google Scholar 

  5. Betancourt A, Morerio P, Regazzoni C S, Rauterberg M (2015) The evolution of first person vision methods: a survey. IEEE Trans Circ Syst Video Technol 25(5):744–760

    Article  Google Scholar 

  6. Qiao S, Tang C, Jin H, et al. (2010) Putmode: prediction of uncertain trajectories in moving objects databases. Appl Intell 33:370–386

    Article  Google Scholar 

  7. Luber M, Stork J A, Tipaldi G D, Arras KO (2010) People tracking with human motion predictions from social forces. In: IEEE International Conference on Robotics and Automation (ICRA)

  8. Musleh B, Garcia F, Otamendi J, Armingol J, Arturo D (2010) Identifying and tracking pedestrians based on sensor fusion and motion stability predictions. Sensors 10(9):8028–8053

    Article  Google Scholar 

  9. Gupta A, Johnson J, Li F F, Savarese S, Alahi A (2018) Social gan: socially acceptable trajectories with generative adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  10. Huang Y, Bin H, Li Z, Mao T, Wang Z (2019) STGAT: Modeling Spatial-temporal interactions for human trajectory prediction. In: IEEE International Conference on Computer Vision (ICCV)

  11. Mainprice J, Hayne R, Berenson D, Arturo D, Arturo D (2016) Goal set inverse optimal control and iterative replanning for predicting human reaching motions in shared workspaces. IEEE Trans Robot 32(4):897–908

    Article  Google Scholar 

  12. Luo Y, Cai P, Bera A, Hsu D, Lee W S, Manocha D (2018) PORCA: Modeling and planning for autonomous driving among many pedestrians. IEEE Robot Autom Lett 3(4):3418–3425

    Article  Google Scholar 

  13. Alahi A, Goel K, Ramanathan V, Robicquet A, Li F F, Savarese S (2016) Social LSTM: human trajectory prediction in crowded spaces. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  14. Sadeghian A, Kosaraju V, Sadeghian A, Hirose N, Rezatofighi H, Savarese S (2019) Sophie: an attentive gan for predicting paths compliant to social and physical constraints. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  15. Xue H, Huynh D Q, Reynolds M (2018) SS-LSTM: A hierarchical lstm model for pedestrian trajectory prediction. In: IEEE Winter Conference on Applications of Computer Vision

  16. Salzmann T, Ivanovic B, Chakravarty P, Pavone M (2020) Trajectron++: multi-agent generative trajectory forecasting with heterogeneous data for control. arXiv:2001.03093

  17. Li J, Ma H, Tomizuka M (2019) Conditional generative neural system for probabilistic trajectory prediction. In: IEEE International Conference on Intelligent Robots and Systems

  18. Chen Y, Liu C, Shi B, Liu M (2020) CoMoGCN: coherent motion aware trajectory prediction with graph representation. arXiv:2005.00754

  19. Chen Y, Liu C, Shi B, Liu M (2020) Robot navigation in crowds by graph convolutional networks with attention learned from human gaze. IEEE Robot Autom Lett 5(2):2754–2761

    Article  Google Scholar 

  20. Kosaraju V, Sadeghian A, Martin-Martin R, Reid I, Savarese S (2019) Social-BiGAT: multimodal trajectory forecasting using bicycle-gan and graph attention networks. arXiv:1907.03395

  21. Ivanovic B, Pavone M (2019) The trajectron: probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In: IEEE International Conference on Computer Vision (ICCV)

  22. Mohamed A, Qian K, Elhoseiny M, Claudel C (2020) Social-STGCNN: a social spatio-temporal graph convolutional neural network for human trajectory prediction. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  23. Keller C G, Gavrila D M (2014) Will the pedestrian cross? a study on pedestrian path prediction. IEEE Trans Intell Transp Syst 15(2):494–506

    Article  Google Scholar 

  24. Schneider N, Gavrila DM (2013) Pedestrian path prediction with recursive bayesian filters: a comparative study. In: 35th German Conference on Pattern Recognition

  25. Pavlovic V, Rehg J M, Maccormick J (2000) Learning switching linear models of human motion. In: Conference and Workshop on Neural Information Processing Systems (NIPS)

  26. Fox E, Sudderth E B, Jordan M I, Willsky A S (2011) Bayesian nonparametric inference of switching dynamic linear models. IEEE Trans Signal Process 54(9):1569–1585

    Article  Google Scholar 

  27. Fleet D, Pajdla T, Schiele B, Tuytelaars T (2014) Context-based pedestrian path prediction. In: the 13th European Conference on Computer Vision (ECCV)

  28. Helbing D, Molnar P (1995) Social force model for pedestrian dynamics. Phys Rev E 51 (5):4282–4286

    Article  Google Scholar 

  29. Bahdanau D, Cho K, Bengio Y (2016) Neural machine translation by jointly learning to align and translate. arXiv:1409.0473v7

  30. Alahi A, Ramanathan V, Li FF (2014), Socially-aware large-scale crowd forecasting. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  31. Yi S, Li H, Wang X (2015) Understanding pedestrian behaviors from stationary crowd groups. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  32. Ellis D, Sommerlade E, Reid I (2009) Modelling pedestrian trajectory patterns with Gaussian processes. In: IEEE International Conference on Computer Vision (ICCV)

  33. Hochreiter S, Schmidhuber J (1997) Long Short-Term memory. Neural Comput 9(7):1735–1780

    Article  Google Scholar 

  34. Karatzoglou A, Jablonski A, Beigl M (2018) A Seq2Seq learning approach for modeling semantic trajectories and predicting the next location. In: the 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems

  35. Kitani KM, Huang DA, Ma WC (2012) Activity forecasting. In: the 12th European Conference on Computer Vision (ECCV)

  36. Goodfellow I J, Pouget-Abadie J, Mirza M (2014) Generative adversarial networks. Adv Neural Inf Process Syst 3(11):2672–2680

    Google Scholar 

  37. Kipf T N, Welling M (2016) Variational graph auto-encoders. arXiv:1611.07308

  38. Amirian J, Hayet J, Pettre J (2019) Social ways: learning multi-Modal distributions of pedestrian trajectories with gans. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  39. Cheng H, Yang W, Sester M, Rosenhahn B (2020) Context conditional variational autoencoder for predicting multi-path trajectories in mixed traffic. arXiv:2002.05966

  40. Yang B, Yan G, Wang P, Chan C, Liu X, Chen Y (2020) TPPO: a novel trajectory predictor with pseudo oracle. arXiv:2002.01852

  41. Wang M, Shi D, Guan N, Zhang T, Li R (2019) Unsupervised pedestrian trajectory prediction with graph neural networks. In: IEEE 31st International Conference on Tools with Artificial Intelligence

  42. Liang J, Jiang L, Murphy K, Yu T, Hauptmann A (2020) The garden of forking paths: towards multi-future trajectory prediction. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  43. Haddadg S, Wu M, Wei H, Yu T, Lam SK (2020) Situation-aware pedestrian trajectory prediction with spatio-temporal attention model. In: 24th Computer Vision Winter Workshop

  44. Liang J, Jiang L, Niebles C, Hauptmann A G, Li F F (2019) Peeking into the future: predicting future person activities and locations in videos. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  45. Zhang L, She Q, Guo P (2019) Stochastic trajectory prediction with social graph network. arXiv:1907.10233

  46. Zhang P, Ouyang W, Xue J, Zheng N (2019) SR-LSTM: State refinement for lstm towards pedestrian trajectory prediction. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  47. Sun J, Jiang Q, Lu C (2020) Recursive social behavior graph for trajectory prediction. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  48. Salzmann T, Ivanovic B, Chakravarty P, Pavone M (2021) Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data. arXiv:2001.03093

  49. Zamboni S, Kefato Z, Girdzijauskas S, Christoffer N, Col L D (2022) Pedestrian trajectory prediction with convolutional neural networks. Pattern Recogn:121

  50. Vaswani A, Shazeer N, Parmar N, Uszkoreit J (2017) Attention Is All You Need. In: Proceedings of the 31st Advances in Neural Information Processing Systems (NIPS)

  51. Devlin J, Chang M, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Human Language Technologies

  52. Katharopoulos A, Vyas A, Pappas N, Fleuret F (2020) Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. arXiv:2006.16236

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (61976039,52172382) and Science and Technology Innovation Fund of Dalian (2021JJ12GX015), and the China Fundamental Research Funds for the Central Universities (DUT20GJ207).

Author information

Authors and Affiliations

  1. Faculty of Vehicle Engineering and Mechanics, School of Automotive Engineering, Dalian University of Technology, Dalian, 116024, China

    Jing Lian, Weiwei Ren, Linhui Li, Yafu Zhou & Bin Zhou

Authors
  1. Jing Lian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weiwei Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linhui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yafu Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bin Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Linhui Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lian, J., Ren, W., Li, L. et al. PTP-STGCN: Pedestrian Trajectory Prediction Based on a Spatio-temporal Graph Convolutional Neural Network. Appl Intell 53, 2862–2878 (2023). https://doi.org/10.1007/s10489-022-03524-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03524-1

Keywords

Search

Navigation