Skip to main content
Springer Nature Link
Log in

Improving the learning of self-driving vehicles based on real driving behavior using deep neural network techniques

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Considering the significant advancements in autonomous vehicle technology, research in this field is of interest to researchers. To drive vehicles autonomously, controlling steer angle, gas hatch, and brakes needs to be learned. The behavioral cloning method is used to imitate humans’ driving behavior. We created a dataset of driving in different routes and conditions, and using the designed model, the output used for controlling the vehicle is obtained. In this paper, the learning of self-driving vehicles based on real driving behavior using deep neural network techniques (LSV-DNN) is proposed. We designed a convolutional network which uses the real driving data obtained through the vehicle’s camera and computer. The response of the driver during driving is recorded in different situations, and by converting the real driver’s driving video to images and transferring the data to an Excel file, obstacle detection is carried out with the best accuracy and speed using the Yolo algorithm version 3. This way, the network learns the response of the driver to obstacles in different locations and the network is trained with the Yolo algorithm version 3 and the output of obstacle detection. Then, it outputs the steer angle and amount of brake, gas, and vehicle acceleration. This study focuses on designing a convolutional network using behavioral cloning and motion planning of autonomous vehicle using a deep learning framework. Neural networks are effective systems for finding relationships between data, modeling, and predict new data or classify data. As a result Neural networks with input real data predict steer angle and speed for autonomous driving. The LSV-DNN is evaluated here via extensive simulations carried out in Python and TensorFlow environment. We evaluated the network error using the loss function. The results confirmed that our scheme is capable of exhibiting high prediction accuracy (exceeding 92.93%). In addition, our proposed scheme has high speed (more than 64.41%), low FPR (less than 6.89%), and low FNR (less than 3.95%), in comparison with the other approaches currently being employed. By comparing other methods which were conducted on the simulator’s data, we obtained good performance results for the designed network on the data from KITTI benchmark, the data collected using a private vehicle, and the data we collected.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Kocić J, Jovičić N, Drndarević V (2019) An End-to-End deep neural network for autonomous driving designed for embedded automotive platforms. Sensor 19(9):2064

    Article  Google Scholar 

  2. Azam S, Munir F, Rafique MA, Sheri AM, Hussain MI, Jeon M (2020). N 2 C: neural network controller design using behavioral cloning. arXiv preprint arXiv:2006.00820

  3. Shahverdy M, Fathy M, Berangi R, Sabokrou M (2020) Driver behavior detection and classification using deep convolutional neural networks. Expert Syst Appl 149:113240

    Article  Google Scholar 

  4. Benjdira B, Khursheed T, Koubaa A, Ammar A, Ouni K (2019) Car detection using unmanned aerial vehicles: comparison between faster r-cnn and yolov3. In: 2019 1st International Conference on Unmanned Vehicle Systems-Oman (UVS). IEEE, pp 1–6

  5. Hatt M, Parmar C, Qi J, El Naqa I (2019) Machine (deep) learning methods for image processing and radiomics. IEEE Trans Radiat Plasma Med Sci 3(2):104–108

    Article  Google Scholar 

  6. Nawrocka D (2018). Machine learning for trading and portfolio management using Python. In: Henderson P, Islam R, Bachman P, Pineau J, Precup D, Meger D (eds) Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018

  7. Patel KK (2020) A simulation environment with reduced reality gap for testing autonomous vehicles (Doctoral dissertation, University of Windsor, Canada)

  8. Xie Y, Le L (2019) Inventors; Equifax Inc, assignee. Dual deep learning architecture for machine-learning systems. United States patent application US 16/141,152

  9. Kanade T, Thorpe C, Whittaker W (1986) Autonomous land vehicle project at CMU. In: Proceedings of the 1986 ACM Fourteenth Annual Conference on Computer Science. ACM, pp 71–80

  10. Wallace RS, Stentz A, Thorpe CE, Moravec HP, Whittaker W, Kanade T (1985) First results in robot road-following. In: IJCAI, pp 1089–1095

  11. Dickmanns ED, Zapp A (1987) Autonomous high speed road vehicle guidance by computer vision. IFAC Proc Vol 20(5):221–226

    Article  Google Scholar 

  12. Thrun S, Montemerlo M, Dahlkamp H, Stavens D, Aron A, Diebel J, Fong P, Gale J, Halpenny M, Hoffmann G, Stanley KL (2006) The robot that won the DARPA Grand Challenge. J Field Robot. 23(9):661–692

    Article  Google Scholar 

  13. Montemerlo M, Thrun S, Dahlkamp H, Stavens D, Strohband S (2006) Winning the DARPA Grand Challenge with an AI robot. In: AAAI 2006, pp 982–987

  14. Buehler M, Iagnemma K, Singh S (eds) (2009) The DARPA urban challenge: autonomous vehicles in city traffic. Springer, New York

    Google Scholar 

  15. Sze V, Chen YH, Yang TJ, Emer JS (2017) Efficient processing of deep neural networks: a tutorial and survey. Proc IEEE 105(12):2295–2329

    Article  Google Scholar 

  16. Cunneen M, Mullins M, Murphy F (2019) Autonomous vehicles and embedded artificial intelligence: the challenges of framing machine driving decisions. Appl Artif Intell 33(8):706–731

    Article  Google Scholar 

  17. Codevilla F, Miiller M, Lo´pez A, Koltun V, Dosovitskiy A (2018) End-to-End driving via conditional imitation learning. In: IEEE International Conference on Robotics and Automation, 2018

  18. Ly AO, Akhloufi MA (2020) Learning to drive by imitation: an overview of deep behavior cloning methods. IEEE Trans Intell Veh

  19. Kelchtermans K, Tuytelaars T (2017) How hard is it to cross the room? Training (recurrent) neural networks to steer a UAV. arXiv preprint arXiv:1702.07600

  20. Hancock PA, Nourbakhsh I, Stewart J (2019) On the future of transportation in an era of automated and autonomous vehicles. Proc Natl Acad Sci 116(16):7684–7691

    Article  Google Scholar 

  21. Xu H, Gao Y, Yu F, Darrell T (2017) End-to-end learning of driving models from large-scale video datasets. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2174–2182

  22. Sadeghi F, Levine S (2016) Cad2rl: real single-image flight without a single real image. arXiv preprint arXiv:1611.04201

  23. Sundermeyer M, Oparin I, Gauvain JL, Freiberg B, Schlüter R, Ney H (2013) Comparison of feedforward and recurrent neural network language models. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, pp 8430–8434

  24. Sundermeyer M, Schlüter R, Ney H (2012) LSTM neural networks for language modeling. In: Thirteenth Annual Conference of the International Speech Communication Association 2012

  25. Pomerleau DA (1989) Alvinn: An autonomous land vehicle in a neural network. In: Advances in neural information processing systems, pp 305–313

  26. Muller U, Ben J, Cosatto E, Flepp B, Cun YL (2006) Off-road obstacle avoidance through end-to-end learning. In: Advances in neural information processing systems, pp 739–746

  27. Bojarski M, Del Testa D, Dworakowski D, Firner B, Flepp B, Goyal P, Jackel LD, Monfort M, Muller U, Zhang J, Zhang X (2016) End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316

  28. Babhulkar A. Self-driving car using udacity’s car simulator environment and trained by deep neural networks

  29. Bach S, Binder A, Montavon G, Klauschen F, Müller KR, Samek W (2015) On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7):e0130140

    Article  Google Scholar 

  30. Binder A, Montavon G, Lapuschkin S, Müller KR, Samek W (2016) Layer-wise relevance propagation for neural networks with local renormalization layers. In: International Conference on Artificial Neural Networks. Springer, Cham, pp 63–71

  31. Lapuschkin S, Binder A, Montavon G, Muller KR, Samek W (2016) Analyzing classifiers: Fisher vectors and deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2912–2920

  32. Codevilla F, Santana E, López AM, Gaidon A (2019) Exploring the limitations of behavior cloning for autonomous driving. In: Proceedings of the IEEE International Conference on Computer Vision, pp 9329–9338

  33. Zhao L, Li S (2020) Object detection algorithm based on improved YOLOv3. Electronics 9(3):537

    Article  Google Scholar 

  34. Arras L, Horn F, Montavon G, Müller KR, Samek W (2016) Explaining predictions of non-linear classifiers in NLP. arXiv preprint arXiv:1606.07298

  35. Huang Y, Chen Y (2020) Autonomous driving with deep learning: a survey of State-of-Art technologies. arXiv preprint arXiv:2006.06091

  36. Nawrocka, D. (2018). Machine learning for trading and portfolio management using Python

  37. Song S, Hu X, Yu J, Bai L, Chen L (2018) Learning a deep motion planning model for autonomous driving. In: IEEE intelligent vehicles symposium

  38. Chowdhuri S, Pankaj T, Zipser K (2019) MultiNet: multi-modal multitask learning for autonomous driving. In: IEEE Winter Conference on Applications of Computer Vision, pp 1496–1504

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Shahr-e-Qods Branch, Islamic Azad University, Tehran, Iran

    Nayereh Zaghari & Seyed Mahdi Jameii

  2. School of Computer Science, Institute for Research in Fundamental Sciences (IPM), Tehran, Iran

    Mahmood Fathy & Mohammad Sabokrou

  3. Department of Computer Engineering, IRAN University of Science and Technology (IUST), Tehran, Iran

    Mohammad Shahverdy

Authors
  1. Nayereh Zaghari
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Mahmood Fathy
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Seyed Mahdi Jameii
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Mohammad Sabokrou
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Mohammad Shahverdy
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Mahmood Fathy.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zaghari, N., Fathy, M., Jameii, S.M. et al. Improving the learning of self-driving vehicles based on real driving behavior using deep neural network techniques. J Supercomput 77, 3752–3794 (2021). https://doi.org/10.1007/s11227-020-03399-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-020-03399-4

Keywords