Skip to main content

Advertisement

Log in

Multi-scale graph-transformer network for trajectory prediction of the autonomous vehicles

  • Review Article
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

The accurate trajectory prediction is a crucial task for the autonomous vehicles that help to plan and fast decision making capability of the system to reach their destination in the complex road scenario with abiding by the traffic rules. For this, autonomous vehicles should have more attention to their goal without affecting the other’s task and maintain their safety from road accidents. With this motivation, we proposed a multi-scale graph-transformer-based attention mechanism that provides the interaction between the road agents with different time instances, because from time to time, few new agents may enter the frame scene, and few may leave the frame scene. Each dynamic obstacles trajectory can be defined as state sequences within an interval of time, where spatial coordinates of dynamic obstacles represented by the each state under the world coordinate frame. We have presented graph-based Multi-scale spatial features with transformer network that achieves significant prediction results compared to other existing methods, and we provide an in-depth analysis of the trained weights for different highways scenarios with transformer and the Long-Short Term Memory. We evaluate our model with three publicly available datasets and achieve state-of-the-art performances as presented in the manuscript. The performance balance is more in favour of our model for sparser datasets compared to the dense datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Chandra R et al (2020) Forecasting trajectory and behavior of road-agents using spectral clustering in graph-LSTMs. IEEE Robot Autom Lett 5(3):4882–4890

    Article  Google Scholar 

  2. Xiong W, Wu L, Alleva F, Droppo J, Huang X, Stolcke A (2018) The microsoft 2017 conversational speech recognition system. In: ICASSP, International conference on acoustics, speech, and signal processing, proceedings vol 2018-April, no. August, pp 5934–5938

  3. Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL HLT 2019—2019 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol.—Proc. Conf., vol 1, no Mlm, pp 4171–4186

  4. Fragkiadaki K, Levine S, Felsen P, Malik J (2015) Recurrent network models for human dynamics. In: Proceedings of the IEEE international conference on computer vision, vol 2015 Inter, pp 4346–4354

  5. Vaswani A et al (2017) Attention is all you need. Adv Neural Inf Process Syst 2017:5999–6009

    Google Scholar 

  6. Luong MT, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: Conf. Proc. - EMNLP 2015 Conf. Empir. Methods Nat. Lang. Process., pp 1412–1421

  7. Plastiras G, Kyrkou C, Theocharides T (2018) Efficient convnet-based object detection for unmanned aerial vehicles by selective tile processing. In: ACM international conference proceeding series

  8. Papageorgiou C, Poggio T (2000) Trainable system for object detection. Int J Comput Vis 38(1):15–33

    Article  Google Scholar 

  9. Lin J, Koch L, Kurowski M, Gehrt JJ, Abel D, Zweigel R (2020) Environment perception and object tracking for autonomous vehicles in a harbor scenario. In: 2020 IEEE 23rd Int. Conf. Intell. Transp. Syst. ITSC 2020, no. 4

  10. Rangesh A, Trivedi MM (2019) No blind spots: full-surround multi-object tracking for autonomous vehicles using cameras and LiDARs. IEEE Trans Intell Veh 4(4):588–599

    Article  Google Scholar 

  11. Zhang P, Ouyang W, Zhang P, Xue J, Zheng N (2019) SR-LSTM: State refinement for lstm towards pedestrian trajectory prediction. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, vol 2019-June, pp 12077–12086

  12. Ivanovic B, Pavone M (2019) The trajectron: probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In: Proceedings of the IEEE international conference on computer vision, vol 2019-Octob, pp 2375–2384

  13. Fisac JF, Bronstein E, Stefansson E, Sadigh D, Sastry SS, Dragan AD (2019) Hierarchical game-theoretic planning for autonomous vehicles. In: Proceedings of international conference on robotics and automation, vol 2019-May, pp 9590–9596

  14. Liu C, Lee S, Varnhagen S, Tseng HE (2017) Path planning for autonomous vehicles using model predictive control. IEEE Intell Veh Symp Proc 5:174–179

    Google Scholar 

  15. Wu M et al (2020) Visual tracking with multiview trajectory prediction. IEEE Trans Image Process 29:8355–8367

    Article  Google Scholar 

  16. Huynh M, Alaghband G (2019) Trajectory prediction by coupling scene-LSTM with human movement LSTM. In: Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol 11844 LNCS, pp 244–259

  17. Dönmez E, Kocamaz AF (2020) Design of mobile robot control infrastructure based on decision trees and adaptive potential area methods. Iran J Sci Technol Trans Electr Eng 44(1):431–448

    Article  Google Scholar 

  18. Dirik M, Kocamaz AF, Dönmez E (2020) Visual servoing based control methods for nonholonomic mobile robot. J Eng Res 8(2):95–113

    Google Scholar 

  19. Okumuş F, Dönmez E, Kocamaz AF (2020) A cloudware architecture for collaboration of multiple agvs in indoor logistics: case study in fabric manufacturing enterprises. Electron 9(12):1–24

    Article  Google Scholar 

  20. Dönmez E, Kocamaz AF, Dirik M (2018) A vision-based real-time mobile robot controller design based on Gaussian function for indoor environment. Arab J Sci Eng 43(12):7127–7142

    Article  Google Scholar 

  21. Gupta A, Johnson J, Fei-Fei L, Savarese S, Alahi A (2018) Social GAN: socially acceptable trajectories with generative adversarial networks. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 2255–2264

  22. Yu C, Ma X, Ren J, Zhao H, Yi S (2020) Spatio-temporal graph transformer networks for pedestrian trajectory prediction

  23. Lee D, Gu Y, Hoang J, Marchetti-Bowick M (2019) Joint interaction and trajectory prediction for autonomous driving using graph neural networks, no. NeurIPS

  24. Sun J, Jiang Q, Lu C (2020) Recursive social behavior graph for trajectory prediction, pp 660–669

  25. Morzy M (2007) Mining frequent trajectories of moving objects for location prediction. In: Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol 4571 LNAI, pp 667–680

  26. Monreale A, Pinelli F, Trasarti R, Giannotti F (2009) WhereNext: a location predictor on trajectory pattern mining. In: Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, pp 637–645

  27. Won JI, Kim SW, Baek JH, Lee J (2009) Trajectory clustering in road network environment. In: 2009 IEEE Symp. Comput. Intell. Data Mining, CIDM 2009 - Proc., pp 299–305

  28. Roh GP, Hwang SW (2010) NNCluster: an efficient clustering algorithm for road network trajectories. In: Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol 5982 LNCS, no. PART 2, pp 47–61

  29. Han B, Liu L, Omiecinski E (2015) Road-network aware trajectory clustering: Integrating locality, flow, and density. IEEE Trans Mob Comput 14(2):416–429

    Article  Google Scholar 

  30. Chen M, Liu Y, Yu X (2015) Predicting next locations with object clustering and trajectory clustering. In: Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol 9078, pp 344–356

  31. Ashbrook D, Starner T (2003) Using GPS to learn significant locations and predict movement across multiple users. Pers Ubiquitous Comput 7(5):275–286

    Article  Google Scholar 

  32. Lv Q, Qiao Y, Ansari N, Liu J, Yang J (2017) Individual mobility prediction at points of interest. IEEE Trans Veh Technol 66(6):5204–5216

    Article  Google Scholar 

  33. Ishikawa Y. From Indexed Spatio-Temporal Datasets. Development, pp 9–16

  34. Gambs S, Killijian MO, Del Prado Cortez MN (2012) Next place prediction using mobility Markov chains. In: Proc. 1st Work. Meas. Privacy, Mobility, MPM’12, pp 0–5

  35. Chandra R, Bhattacharya U, Bera A, Di Manocha R (2019) Traphic: trajectory prediction in dense and heterogeneous traffic using weighted interactions. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, vol 2019, pp 8475–8484

  36. Carrasco S, Llorca DF, Sotelo MÁ (2021) SCOUT: Socially-COnsistent and UndersTandable graph attention network for trajectory prediction of vehicles and VRUs

  37. Gao J et al (2020) VectorNet: encoding HD maps and agent dynamics from vectorized representation. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 11522–11530

  38. Kim B et al (2021) LaPred: lane-aware prediction of multi-modal future trajectories of dynamic agents

  39. Park SH et al (2020) Diverse and admissible trajectory forecasting through multimodal context understanding, Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol 12356 LNCS, pp 282–298

  40. Fang L, Jiang Q, Shi J, Zhou B (2020) TPNet: trajectory proposal network for motion prediction. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 6796–6805

  41. Luo C, Sun L, Dabiri D, Yuille A (2020) Probabilistic multi-modal trajectory prediction with lane attention for autonomous vehicles. IEEE Int Conf Intell Robot Syst 56:2370–2376

    Google Scholar 

  42. He H, Dai H, Wang N (2020) UST: unifying spatio-temporal context for trajectory prediction in autonomous driving. IEEE Int Conf Intell Robot Syst 56:5962–5969

    Google Scholar 

  43. Wang J et al (2021) “F-NET: fusion neural network for vehicle trajectory prediction in autonomous driving, vol 1. Peking University, Beijing, pp 4095–4099

    Google Scholar 

  44. Becker S, Hug R, Hübner W, Arens M (2018) An evaluation of trajectory prediction approaches and notes on the TrajNet Benchmark

  45. Becker S, Hug R, Hübner W, Arens M (2019) RED: a simple but effective baseline predictor for the TrajNet benchmark. In: Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol 11131 LNCS, pp 138–153

  46. Scholler C, Aravantinos V, Lay F, Knoll A (2020) What the constant velocity model can teach us about pedestrian motion prediction. IEEE Robot Autom Lett 5(2):1696–1703

    Article  Google Scholar 

  47. Hammond DK, Vandergheynst P, Gribonval R (2011) Wavelets on graphs via spectral graph theory. Appl Comput Harmon Anal 30(2):129–150

    Article  MathSciNet  Google Scholar 

  48. Badue C et al (2017) 1901.04407-1

  49. Ma Y, Zhu X, Zhang S, Yang R, Wang W, Manocha D (2009) TrafficPredict: trajectory prediction for heterogeneous traffic-agents, no. Kalman 1960

  50. Chen L (2020) One thousand and one hours: self-driving motion prediction dataset, no. CoRL 2020, pp 1–10

  51. Chang M et al. Argoverse : 3D tracking and forecasting with rich maps

  52. Li X, Ying X, Chuah MC (2019) GRIP++: enhanced graph-based interaction-aware trajectory prediction for autonomous driving

  53. Li X, Ying X, Chuah MC (2019) GRIP: graph-based interaction-aware trajectory prediction, pp 3960–3966

  54. Julka S, Sowrirajan V, Schloetterer J, Granitzer M (2021) Conditional generative adversarial networks for speed control in trajectory simulation

  55. Kim G, Kim D, Ahn Y, Huh K (2021) Hybrid approach for vehicle trajectory prediction using weighted integration of multiple models. IEEE Access 56:1

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Divya Singh.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Singh, D., Srivastava, R. Multi-scale graph-transformer network for trajectory prediction of the autonomous vehicles. Intel Serv Robotics 15, 307–320 (2022). https://doi.org/10.1007/s11370-022-00422-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-022-00422-w

Keywords

Navigation