Skip to main content
SpringerLink
Log in

Automatic Bi-LSTM Architecture Search Using Bayesian Optimisation for Vehicle Activity Recognition

  • Conference paper
  • First Online:
Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1815))

  • 103 Accesses

Abstract

This paper presents a novel method to find optimal Bidirectional Long-Short Term Memory Neural Network (Bi-LSTM) using Bayesian Optimisation method for vehicle trajectory classification. We extend our previous approach to be able to classify a larger number of vehicle trajectories collected from different sources in a single Bi-LSTM network. We also explored the use of deep learning visual explainability by highlighting the parts of the activity (or trajectory) contribute to the classification decision of the network. In particular, Qualitative Trajectory Calculus (QTC), spatio-temporal calculus, method is used to encode the relative movement between vehicles as a trajectory of QTC states. We then develop a Bi-LSTM network (called VNet) to classify QTC trajectories that represent vehicle pairwise activities. Existing Bi-LSTM networks for vehicle activity analysis are manually designed without considering the optimisation of the whole architecture nor its trainable hyperparameters. Therefore, we adapt Bayesian Optimisation method to search for an optimal Bi-LSTM architecture for classifying QTC trajectories of vehicle interaction. To test the validity of the proposed VNet, four datasets of 8237 trajectories of 9 unique vehicle activities in different traffic scenarios are used. We further compare our VNet model’s performance with the state-of-the-art methods. The results on the combined dataset (accuracy of 98.21%) showed that the proposed method generates light and most robust Bi-LSTM model. We also demonstrate that Activation Map is a promising approach for visualising the Bi-LSTM model decisions for vehicle activity recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ahmed, M., Du, H., AlZoubi, A.: An ENAS based approach for constructing deep learning models for breast cancer recognition from ultrasound images. arXiv preprint arXiv:2005.13695 (2020)

  2. Ahmed, S.A., Dogra, D.P., Kar, S., Roy, P.P.: Trajectory-based surveillance analysis: a survey. IEEE Trans. Circuits Syst. Video Technol. 29(7), 1985–1997 (2018)

    Article  Google Scholar 

  3. Altché, F., de La Fortelle, A.: An LSTM network for highway trajectory prediction. In: 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 353–359 (2017). https://doi.org/10.1109/ITSC.2017.8317913

  4. AlZoubi, A., Al-Diri, B., Pike, T., Kleinhappel, T., Dickinson, P.: Pair-activity analysis from video using qualitative trajectory calculus. IEEE Trans. Circuits Syst. Video Technol. 28(8), 1850–1863 (2017)

    Article  Google Scholar 

  5. Alzoubi, A., Nam, D.: Vehicle Obstacle Interaction Dataset (VOIDataset), October 2018. https://doi.org/10.17862/cranfield.rd.6270233.v2

  6. AlZoubi, A., Nam, D.: Vehicle activity recognition using DCNN. In: Cláudio, A.P., et al. (eds.) VISIGRAPP 2019. CCIS, vol. 1182, pp. 566–588. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-41590-7_24

    Chapter  Google Scholar 

  7. AlZoubi, A., Nam, D.: Vehicle activity recognition using mapped QTC trajectories. In: Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 5: VISAPP, pp. 27–38. INSTICC, SciTePress (2019). https://doi.org/10.5220/0007307600270038

  8. Beernaerts, J., De Baets, B., Lenoir, M., De Mey, K., Van de Weghe, N.: Analysing team formations in football with the static qualitative trajectory calculus. In: Proceedings of the 7th icSPORTS International Conference on Sports Science Research and Technology Support, Seville, Spain, pp. 20–21 (2018)

    Google Scholar 

  9. Berndt, H., Dietmayer, K.: Driver intention inference with vehicle onboard sensors. In: 2009 IEEE International Conference on Vehicular Electronics and Safety (ICVES), pp. 102–107. IEEE (2009)

    Google Scholar 

  10. Chavoshi, S.H., et al.: Knowledge discovery in choreographic data using relative motion matrices and dynamic time warping. Appl. Geogr. 47, 111–124 (2014)

    Article  Google Scholar 

  11. Chavoshi, S.H., De Baets, B., Neutens, T., De Tré, G., Van de Weghe, N.: Exploring dance movement data using sequence alignment methods. PLoS ONE 10(7), e0132452 (2015)

    Article  Google Scholar 

  12. Chen, L., Özsu, M.T., Oria, V.: Robust and fast similarity search for moving object trajectories. In: Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, pp. 491–502 (2005)

    Google Scholar 

  13. Dahl, G.E., Sainath, T.N., Hinton, G.E.: Improving deep neural networks for LVCSR using rectified linear units and dropout. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8609–8613. IEEE (2013)

    Google Scholar 

  14. Deo, N., Rangesh, A., Trivedi, M.M.: How would surround vehicles move? A unified framework for maneuver classification and motion prediction. IEEE Trans. Intell. Veh. 3(2), 129–140 (2018)

    Article  Google Scholar 

  15. Deo, N., Trivedi, M.M.: Convolutional social pooling for vehicle trajectory prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1468–1476 (2018)

    Google Scholar 

  16. Deo, N., Trivedi, M.M.: Multi-modal trajectory prediction of surrounding vehicles with maneuver based LSTMS. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1179–1184. IEEE (2018)

    Google Scholar 

  17. Dondrup, C., Bellotto, N., Hanheide, M., Eder, K., Leonards, U.: A computational model of human-robot spatial interactions based on a qualitative trajectory calculus. Robotics 4(1), 63–102 (2015)

    Article  Google Scholar 

  18. Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning. TSSCML, pp. 3–33. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05318-5_1

    Chapter  Google Scholar 

  19. Framing, C.E., Heßeler, F.J., Abel, D.: Infrastructure-based vehicle maneuver estimation with intersection-specific models. In: 2018 26th Mediterranean Conference on Control and Automation (MED), pp. 253–258. IEEE (2018)

    Google Scholar 

  20. Frazier, P.I.: A tutorial on Bayesian optimization (2018)

    Google Scholar 

  21. Hanheide, M., Peters, A., Bellotto, N.: Analysis of human-robot spatial behaviour applying a qualitative trajectory calculus. In: 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, pp. 689–694. IEEE (2012)

    Google Scholar 

  22. Kaselimi, M., Doulamis, N., Doulamis, A., Voulodimos, A., Protopapadakis, E.: Bayesian-optimized bidirectional LSTM regression model for non-intrusive load monitoring. In: ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2747–2751. IEEE (2019)

    Google Scholar 

  23. Khosroshahi, A., Ohn-Bar, E., Trivedi, M.M.: Surround vehicles trajectory analysis with recurrent neural networks. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pp. 2267–2272. IEEE (2016)

    Google Scholar 

  24. Kim, B., Kang, C.M., Kim, J., Lee, S.H., Chung, C.C., Choi, J.W.: Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network. In: 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 399–404 (2017). https://doi.org/10.1109/ITSC.2017.8317943

  25. Kim, T.Y., Cho, S.B.: Particle swarm optimization-based CNN-LSTM networks for forecasting energy consumption. In: 2019 IEEE Congress on Evolutionary Computation (CEC), pp. 1510–1516. IEEE (2019)

    Google Scholar 

  26. Krajewski, R., Bock, J., Kloeker, L., Eckstein, L.: The highD dataset: a drone dataset of naturalistic vehicle trajectories on German highways for validation of highly automated driving systems. In: 2018 IEEE 21st International Conference on Intelligent Transportation Systems (ITSC) (2018)

    Google Scholar 

  27. Lefèvre, S., Laugier, C., Ibañez-Guzmán, J.: Exploiting map information for driver intention estimation at road intersections. In: 2011 IEEE Intelligent Vehicles Symposium (IV), pp. 583–588. IEEE (2011)

    Google Scholar 

  28. Lenik, P., Krzeszowski, T., Przednowek, K., Lenik, J.: The analysis of basketball free throw trajectory using PSO algorithm. In: icSPORTS, pp. 250–256 (2015)

    Google Scholar 

  29. Lin, W., Chu, H., Wu, J., Sheng, B., Chen, Z.: A heat-map-based algorithm for recognizing group activities in videos. IEEE Trans. Circuits Syst. Video Technol. 23(11), 1980–1992 (2013)

    Article  Google Scholar 

  30. Lin, W., Sun, M.T., Poovendran, R., Zhang, Z.: Group event detection with a varying number of group members for video surveillance. IEEE Trans. Circuits Syst. Video Technol. 20(8), 1057–1067 (2010)

    Article  Google Scholar 

  31. Melis, G., Dyer, C., Blunsom, P.: On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589 (2017)

  32. Ni, B., Yan, S., Kassim, A.: Recognizing human group activities with localized causalities. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1470–1477. IEEE (2009)

    Google Scholar 

  33. Panzner, M., Cimiano, P.: Comparing hidden Markov models and long short term memory neural networks for learning action representations. In: Pardalos, P.M., Conca, P., Giuffrida, G., Nicosia, G. (eds.) MOD 2016. LNCS, vol. 10122, pp. 94–105. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-51469-7_8

    Chapter  Google Scholar 

  34. Park, S.H., Kim, B., Kang, C.M., Chung, C.C., Choi, J.W.: Sequence-to-sequence prediction of vehicle trajectory via LSTM encoder-decoder architecture. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1672–1678. IEEE (2018)

    Google Scholar 

  35. Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameters sharing. In: International Conference on Machine Learning, pp. 4095–4104. PMLR (2018)

    Google Scholar 

  36. Phillips, D.J., Wheeler, T.A., Kochenderfer, M.J.: Generalizable intention prediction of human drivers at intersections. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1665–1670 (2017). https://doi.org/10.1109/IVS.2017.7995948

  37. Radhakrishnan, R., AlZoubi, A.: Vehicle pair activity classification using QTC and long short term memory neural network. In: VISIGRAPP (5: VISAPP), pp. 236–247 (2022)

    Google Scholar 

  38. Reimers, N., Gurevych, I.: Optimal hyperparameters for deep LSTM-networks for sequence labeling tasks. arXiv preprint arXiv:1707.06799 (2017)

  39. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)

    Google Scholar 

  40. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  41. Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems, vol. 25 (2012)

    Google Scholar 

  42. Snoek, J., et al.: Scalable Bayesian optimization using deep neural networks. In: International Conference on Machine Learning, pp. 2171–2180. PMLR (2015)

    Google Scholar 

  43. Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)

    Google Scholar 

  44. U.S. Department of Transportation Federal Highway Administration: Next generation simulation (NGSIM) vehicle trajectories and supporting data (2016)

    Google Scholar 

  45. Van Der Westhuizen, J., Lasenby, J.: Techniques for visualizing LSTMS applied to electrocardiograms. arXiv preprint arXiv:1705.08153 (2017)

  46. Van de Weghe, N.: Representing and reasoning about moving objects: a qualitative approach. Ph.D. thesis, Ghent University (2004)

    Google Scholar 

  47. Yang, T., Li, B., Xun, Q.: LSTM-attention-embedding model-based day-ahead prediction of photovoltaic power output using Bayesian optimization. IEEE Access 7, 171471–171484 (2019)

    Article  Google Scholar 

  48. Zhou, Y., Yan, S., Huang, T.S.: Pair-activity classification by bi-trajectories analysis. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)

    Google Scholar 

  49. Zyner, A., Worrall, S., Nebot, E.: A recurrent neural network solution for predicting driver intention at unsignalized intersections. IEEE Robot. Autom. Lett. 3(3), 1759–1764 (2018). https://doi.org/10.1109/LRA.2018.2805314

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing, The University of Buckingham, Buckingham, U.K.

    Rahulan Radhakrishnan

  2. School of Computing and Engineering, University of Derby, Derby, U.K.

    Alaa AlZoubi

Authors
  1. Rahulan Radhakrishnan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alaa AlZoubi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rahulan Radhakrishnan .

Editor information

Editors and Affiliations

  1. University of Porto, Porto, Portugal

    A. Augusto de Sousa

  2. University of Warwick, Coventry, UK

    Kurt Debattista

  3. Mines ParisTech, Paris, France

    Alexis Paljic

  4. Bentley University, Waltham, USA

    Mounia Ziat

  5. French Civil Aviation University (ENAC), Toulouse, France

    Christophe Hurter

  6. Monash University, Melbourne, VIC, Australia

    Helen Purchase

  7. Department of Mathematics, University of Catania, Catania, Italy

    Giovanni Maria Farinella

  8. Computer Vision Center, University of Barcelona, Barcelona, Spain

    Petia Radeva

  9. IRISA, University of Rennes 1, Rennes, France

    Kadi Bouatouch

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Radhakrishnan, R., AlZoubi, A. (2023). Automatic Bi-LSTM Architecture Search Using Bayesian Optimisation for Vehicle Activity Recognition. In: de Sousa, A.A., et al. Computer Vision, Imaging and Computer Graphics Theory and Applications. VISIGRAPP 2022. Communications in Computer and Information Science, vol 1815. Springer, Cham. https://doi.org/10.1007/978-3-031-45725-8_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-45725-8_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-45724-1

  • Online ISBN: 978-3-031-45725-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation