Skip to main content
SpringerLink
Log in

Exploring Complex Dependencies for Multi-modal Semantic Trajectory Prediction

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Multi-modal semantic trajectory prediction is of great importance for location-based applications. However, predicting trajectory is not trivial facing three challenges: (1) It is difficult to integrate useful information from multi-modal and heterogeneous data in different granularity for effective feature fusion; (2) All kinds of dependencies existing in multi-modal semantic trajectories are closely coupled and dynamically evolved, forming complex dependencies for which are difficult to quantify; (3) During the model training, the distribution of each modal feature shifts in different directions, resulting in the distortion of dependencies, which is accompanied by slow convergence and inaccurate predictions. In this paper, the Complex Dependencies Auto-learning Prediction Model (CDAPM) is proposed to solve these problems. First, the effective and robust representation of each points is obtained by jointly embedding multi-modal information. Then, the dependencies attention module is proposed to calculate the dependencies weight matrix and auto-learn the contribution of each point. Also, it solves the problem of long-term dependency effectively. Position Encoding and LSTM are used to enhance the time relationship of trajectory. Finally, Mode Normalization is designed to maintain prediction accuracy by preventing the distortion of dependencies and significantly accelerate the convergence speed. Experiments on two real data sets show that CDAPM outperforms the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Availability of data and material

The data sets can be downloaded from http://github.com/liu-jie-cumt/CDAPM.

Code availability

The source code can be downloaded from http://github.com/liu-jie-cumt/CDAPM.

References

  1. Altaf B, Yu L, Zhang X (2018) Spatio-temporal attention based recurrent neural network for next location prediction. BigData, pp 937–942

  2. Asahara A, Maruyama K, Sato A, Seto K (2011) Pedestrian-movement prediction based on mixed Markov-Chain model. In: Proceedings of the 19th ACM SIGSPATIAL international conference on advances in geographic information systems, pp 25–33 (2011)

  3. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. In: International conference on learning representations (2014)

  4. Begleiter R, El-Yaniv R, Yona G (2004) On prediction using variable order Markov models. J Artif Intell Res 22:385–421

    Article  MathSciNet  Google Scholar 

  5. Burbey I, Martin TL (2008) Predicting future locations using prediction-by-partial-match. In: Proceedings of the first ACM international workshop on Mobile entity localization and tracking in GPS-less environments, pp 1–6 (2008)

  6. Chen M, Zuo Y, Jia X, Liu Y, Yu X, Zheng K (2020) Cem: a convolutional embedding model for predicting next locations. IEEE Trans Intell Transp Syst 22(6):3349–3358

  7. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555

  8. Deecke L, Murray I, Bilen H (2018) Mode normalization. arXiv preprint arXiv:1810.05466

  9. Duong-Trung N, Schilling N, Schmidt-Thieme L (2016) Near real-time geolocation prediction in twitter streams via matrix factorization based regression. In: ACM international conference on information and knowledge management (2016)

  10. Feng J, Li Y, Zhang C, Sun F, Meng F, Guo A, Jin D (2018) Deepmove: Predicting human mobility with attentional recurrent networks. In: WWW’18: the web conference 2018 Lyon France April, 2018 pp 1459–1468

  11. Fu X, Jiang Y, Lu G, Wang J, Huang D, Yao D (2014) Probabilistic trajectory prediction in intelligent driving. IFAC Proceedings Volumes 47(3):2664–2672

    Article  Google Scholar 

  12. Gao H, Tang J, Liu H (2012) Mobile location prediction in spatio-temporal context. In: Nokia mobile data challenge workshop, vol 41, pp 1–4

  13. Goli SA, Far BH, Fapojuwo AO (2018) Vehicle trajectory prediction with Gaussian process regression in connected vehicle environment. In: 2018 IEEE intelligent vehicles symposium (IV). IEEE, pp 550–555

  14. Han Q, Lu D, Zhang K, Du X, Guizani M (2019) A prediction method for destination based on the semantic transfer model. IEEE Access 7:73756–73763

    Article  Google Scholar 

  15. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167

  16. Ishikawa Y, Tsukamoto Y, Kitagawa H (2004) Extracting mobility statistics from indexed spatio-temporal datasets. In: STDBM, pp 9–16

  17. Jaccard P (1912) The distribution of the flora in the alpine zone. 1. New Phytologist 11(2):37–50

    Article  Google Scholar 

  18. Karatzoglou A, Beigl M (2019) Semantic-enhanced learning (sel) on artificial neural networks using the example of semantic location prediction. In: Proceedings of the 27th ACM SIGSPATIAL international conference on advances in geographic information systems, pp 448–451 (2019)

  19. Karatzoglou A, Köhler D, Beigl M (2018) Semantic-enhanced multi-dimensional Markov Chains on semantic trajectories for predicting future locations. Sensors 18(10):3582

    Article  Google Scholar 

  20. Krishnamurthy R, Kapanipathi P, Sheth AP, Thirunarayan K (2015) Knowledge enabled approach to predict the location of twitter users. In: European semantic web conference. Springer, pp 187–201

  21. Laurila JK, Gatica-Perez D, Aad I, Bornet O, Do TMT, Dousse O, Eberle J, Miettinen M (2012) The mobile data challenge: Big data for mobile computing research

  22. Li F, Li Q, Li Z, Huang Z, Chang X, Xia J (2019) A personal location prediction method based on individual trajectory and group trajectory. IEEE Access 7:92850–92860

    Article  Google Scholar 

  23. Lipton ZC, Berkowitz J, Elkan C (2015) A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv:1506.00019

  24. Liu Q, Wu S, Wang L, Tan T (2016) Predicting the next location: a recurrent model with spatial and temporal contexts. In: Thirtieth AAAI conference on artificial intelligence

  25. Liu S, Wang L (2018) A self-adaptive point-of-interest recommendation algorithm based on a multi-order Markov model. Future Gener Comput Syst 89:506–514

    Article  Google Scholar 

  26. Liu Y, Seah HS (2015) Points of interest recommendation from gps trajectories. Int J Geogr Inf Sci 29(6):953–979

    Article  Google Scholar 

  27. Palangi H, Deng L, Shen Y, Gao J, He X, Chen J, Song X, Ward R (2016) Deep sentence embedding using long short-term memory networks: analysis and application to information retrieval. IEEE/ACM Trans Audio Speech Lang Process 24(4):694–707

    Article  Google Scholar 

  28. Pang Y, Liu Y (2020) Probabilistic aircraft trajectory prediction considering weather uncertainties using dropout as Bayesian approximate variational inference. In: AIAA Scitech 2020 Forum, p 1413

  29. Rathore P, Kumar D, Rajasegarar S, Palaniswami M, Bezdek JC (2019) A scalable framework for trajectory prediction. IEEE Trans Intell Transp Syst 20(10):3860–3874

    Article  Google Scholar 

  30. Sadr H, Pedram MM, Teshnehlab M (2019) A robust sentiment analysis method based on sequential combination of convolutional and recursive neural networks. Neural Process Lett 50(3):2745–2761

  31. Sadr H, Pedram MM, Teshnehlab M (2020) Multi-view deep network: a deep model based on learning features from heterogeneous neural networks for sentiment analysis. IEEE Access 8:86984–86997

  32. Schmidhuber J, Hochreiter S (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  33. Simmons R, Browning B, Zhang Y, Sadekar V (2006) Learning to predict driver route and destination intent. In: 2006 IEEE intelligent transportation systems conference. IEEE, pp 127–132

  34. Song X, Kanasugi H, Shibasaki R (2016) Deeptransport: prediction and simulation of human mobility and transportation mode at a citywide level. IJCAI 16:2618–2624

    Google Scholar 

  35. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Advances in neural information processing systems, pp 3104–3112

  36. Sutskever I, Vinyals O, Le VQ (2014) Sequence to sequence learning with neural networks. In: NIPS, pp 3104–3112

  37. Tversky A (1977) Features of similarity. Psychol Rev 84(4):327

    Article  Google Scholar 

  38. Visin F, Kastner K, Cho K, Matteucci M, Courville A, Bengio Y (2015) A recurrent neural network based alternative to convolutional networks. arXiv preprint arXiv:1505.00393

  39. Wang H, Yang Z, Shi Y (2019) Next location prediction based on an Adaboost-Markov model of mobile users. Sensors 19(6):1475

    Article  Google Scholar 

  40. Wiest J, Höffken M, Kreßel U, Dietmayer K (2012) Probabilistic trajectory prediction with gaussian mixture models. In: 2012 IEEE intelligent vehicles symposium. IEEE, pp 141–146

  41. Wiest J, Kunz F, Kreßel U, Dietmaye, K (2013) Incorporating categorical information for enhanced probabilistic trajectory prediction. In: 2013 12th international conference on machine learning and applications, vol 1. IEEE, pp 402–407

  42. Xiao Y, Nian Q (2020) Vehicle location prediction based on spatiotemporal feature transformation and hybrid lstm neural network. Information 11(2):84

    Article  Google Scholar 

  43. Yang J, Xu J, Xu M, Zheng N, Chen Y (2014) Predicting next location using a variable order Markov model. In: Proceedings of the 5th ACM SIGSPATIAL international workshop on GeoStreaming, pp 37–42

  44. Yao D, Zhang C, Huang J, Bi, J (2017) Serm: a recurrent model for next location prediction in semantic trajectories. In: Proceedings of the 2017 ACM on conference on information and knowledge management, pp 2411–2414

  45. Zeng J, Tang H, Wu Y, Liu L, Hirokawa S (2019) Predict the next location from trajectory based on spatiotemporal sequence. In: 2019 8th international congress on advanced applied informatics (IIAI-AAI). IEEE, pp 109-114 (2019)

  46. Zhang C, Han J, Shou L, Lu J, Porta FLT (2014) Splitter: mining fine-grained sequential patterns in semantic trajectories. PVLDB

  47. Zhang C, Zhang K, Yuan Q, Zhang L, Hanratty T, Han J (2016) Gmove: Group-level mobility modeling using geo-tagged social media. KDD

  48. Zhang R, Guo J, Jiang H, Xie P, Wang C (2019) Multi-task learning for location prediction with deep multi-model ensembles. In: 2019 IEEE 21st international conference on high performance computing and communications; IEEE 17th international conference on smart city; IEEE 5th international conference on data science and systems (HPCC/SmartCity/DSS). IEEE, pp 1093–1100

Download references

Acknowledgements

We thank anonymous reviewers for valuable suggestions.

Funding

This work was supported in part by “The Double-First-Rate Special Fund for Construction of China University of Mining and Technology, No. 2018ZZCX14.” The funder had no role in study design, data collection and preparation of the manuscript.

Author information

Authors and Affiliations

  1. Engineering Research Center of Mine Digitalization (Ministry of Education), China University of Mining and Technology, Xuzhou, 221116, China

    Jie Liu, Lei Zhang, Shaojie Zhu, Bailong Liu & Zhizheng Liang

  2. School of Computer Science, China University of Mining and Technology, Xuzhou, 221116, China

    Jie Liu, Lei Zhang, Shaojie Zhu, Bailong Liu & Zhizheng Liang

  3. Operating Branch, Ningbo Rail Transit Group Co., LTD, Ningbo, 315100, China

    Susong Yang

Authors
  1. Jie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shaojie Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bailong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhizheng Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Susong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JL and LZ conceived the prediction method, implemented the experiments, conducted the experimental result analysis, and wrote the paper; SZ and BL gathered data and performed experiments. BL, ZL and SY revised the paper. All authors have read and approved the final paper.

Corresponding author

Correspondence to Lei Zhang.

Ethics declarations

Conflict of interest

The authors declare no potential conflicts of interest with respect to the research, authorship, and publication of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, J., Zhang, L., Zhu, S. et al. Exploring Complex Dependencies for Multi-modal Semantic Trajectory Prediction. Neural Process Lett 54, 961–985 (2022). https://doi.org/10.1007/s11063-021-10666-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-021-10666-9

Keywords

Search

Navigation