Skip to main content
SpringerLink
Log in

Anticipating Autonomous Vehicle Driving based on Multi-Modal Multiple Motion Tasks Network

  • Regular paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Recently, research concerning autonomous self-driving vehicles has become very popular. In autonomous vehicles (AVs), different decision-making and learning architectures have been proposed to predict multiple tasks (MTs) or MTs from various datasets or to improve performance among different MTs. In this paper, a novel and unified multitask learning framework, called Multi-Modal DenseNet (M2-DenseNet), is proposed to predict MTs in a single network in which three long short-term memory units act as the output (MTs). Accordingly, the proposed M2-DenseNet can predict three different motion decision-making tasks, i.e., the steering angle, speed, and throttle, to control AV driving. Moreover, M2-DenseNet can greatly reduce the time complexity (e.g., to less than 5 ms) because the different prediction tasks can be predicted simultaneously. We conduct comprehensive experiments with the lane-keeping task based on two control mechanisms using the proposed M2-DenseNet and other existing methods to evaluate the performance. The experiments demonstrate that M2-DenseNet significantly outperforms other state-of-the-art methods with the accuracies of the three control tasks being approximately 98%, 99%, and 98%, respectively. The mean squared error between the predicted value and the ground truth is reported in the experiments, with values for the steering angle, speed, and throttle of 0.0250, 0.0210, and 0.0242, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Code Availability

The code use to obtain results during the study are available the corresponding author oa reasonable request.

References

  1. Chen, Z., Huang, X.: End-to-end learning for lane keeping of self-driving cars. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1856–1860, IEEE (2017)

  2. Haavaldsen, H., Aasboe, M., Lindseth, F.: Autonomous vehicle control: End-to-end learning in simulated urban environments. In: Symposium of the Norwegian AI Society, Springer, pp. 40–51 (2019)

  3. Eraqi, H.M., Moustafa, M.N., Honer, J.: End-to-end deep learning for steering autonomous vehicles considering temporal dependencies. arXiv:1710.03804 (2017)

  4. Chen, Y., Praveen, P., Priyantha, M., Muelling, K., Dolan, J.: Learning on-road visual control for self-driving vehicles with auxiliary tasks. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 331–338, IEEE (2019)

  5. Yang, S., Wang, W., Liu, C., Deng, W., Hedrick, J.K.: Feature analysis and selection for training an end-to-end autonomous vehicle controller using deep learning approach. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1033–1038, IEEE (2017)

  6. Fernandez, N.: Two-stream convolutional networks for end-to-end learning of self-driving cars. arXiv:1811.05785 (2018)

  7. Gu, Z., Li, Z., Di, X., Shi, R.: An LSTM-based autonomous driving model using a waymo open dataset. Appl. Sci. 10(6), 2046 (2020)

    Article  Google Scholar 

  8. Hu, X., Tang, B., Chen, L., Song, S., Tong, X: Learning a Deep Cascaded Neural Network for Multiple Motion Commands Prediction in Autonomous Driving. IEEE Transactions on Intelligent Transportation Systems (2020)

  9. Yang, S., Wang, W., Liu, C., Deng, W.: Scene understanding in deep learning-based end-to-end controllers for autonomous vehicles. IEEE Transactions on Systems, Man, and Cybernetics: Systems 49(1), 53–63 (2018)

    Article  Google Scholar 

  10. Song, S., Hu, X., Yu, J., Bai, L., Chen, L.: Learning a deep motion planning model for autonomous driving. In: 2018 Intelligent Vehicles Symposium (IV), pp. 1137–1142, IEEE (2018)

  11. Valiente, R., Zaman, M., Ozer, S., Fallah, Y.P.: Controlling steering angle for cooperative self-driving vehicles utilizing cnn and lstm-based deep networks. In: 2019 IEEE Intelligent Vehicles Symposium (IV), IEEE, pp. 2423–2428 (2019)

  12. Zhang, Z., Ohya, J.: Movement control with vehicle-to-vehicle communication by using end-to-end deep learning for autonomous driving. In: ICPRAM, pp. 377–385 (2021)

  13. Wu, T., Luo, A., Huang, R., Cheng, H., Zhao, Y.: End-to-end driving model for steering control of autonomous vehicles with future spatiotemporal features. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 950–955, IEEE (2019)

  14. Li, L., Ota, K., Dong, M.: Humanlike driving: empirical decision-making system for autonomous vehicles. IEEE Trans. Veh. Technol. 67(8), 6814–6823 (2018)

    Article  Google Scholar 

  15. Li, X., Sun, Z., Cao, D., He, Z., Zhu, Q.: Real-time trajectory planning for autonomous urban driving: Framework, algorithms, and verifications. IEEE/ASME Transactions on Mechatronics 21(2), 740–753 (2015)

    Article  Google Scholar 

  16. Li, J., Mei, X., Prokhorov, D., Tao, D.: Deep neural network for structural prediction and lane detection in traffic scene. IEEE Transactions on Neural Networks and Learning Systems 28(3), 690–703 (2016)

    Article  Google Scholar 

  17. Paden, B., Čáp, M., Yong, S. Z., Yershov, D., Frazzoli, E.: A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on Intelligent Vehicles 1(1), 33–55 (2016)

    Article  Google Scholar 

  18. Claussmann, L., Revilloud, M., Gruyer, D., Glaser, S.: A Review of Motion Planning for Highway Autonomous Driving. IEEE Transactions on Intelligent Transportation Systems (2019)

  19. Hubschneider, C., et al.: Integrating end-to-end learned steering into probabilistic autonomous driving. In: 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 1–7. IEEE (2017)

  20. Jiang, H., Chang, L., Li, Q., Chen, D.: Deep transfer learning enable end-to-end steering angles prediction for self-driving car. In: IEEE Intelligent Vehicles Symposium (IV): pp. 405–412. IEEE (2020)

  21. Smolyakov, M., Frolov, A., Volkov, V., Stelmashchuk, I.: Self-driving car steering angle prediction based on deep neural network an example Of CarND Udacity simulator. In: 2018 IEEE 12th International Conference on Application of Information and Communication Technologies (AICT), pp. 1–5, IEEE (2018)

  22. del Egio, J., Bergasa, L.M., Romera, E., Huélamo, C.G., Araluce, J., Barea, R.: Self-driving a car in simulation through a CNN. In: Workshop of Physical Agents, Springer, pp. 31–43 (2018)

  23. Chen, L., Wang, Q., Lu, X., Cao, D., Wang, F.-Y.: Learning driving models from parallel end-to-end driving data set. Proc. IEEE 108(2), 262–273 (2019)

    Article  Google Scholar 

  24. Kim, J., Park, C.: End-to-end ego lane estimation based on sequential transfer learning for self-driving cars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 30–38 (2017)

  25. Chen, L., Hu, X., Tang, B., Cao, D.: Parallel motion planning: learning a deep planning model against emergencies. IEEE Intell. Transp. Syst. Mag. 11(1), 36–41 (2018)

    Article  Google Scholar 

  26. Chen, L., Hu, X., Tian, W., Wang, H., Cao, D., Wang, F.-Y.: Parallel planning: a new motion planning framework for autonomous driving. IEEE/CAA Journal of Automatica Sinica 6(1), 236–246 (2018)

    Article  Google Scholar 

  27. Yuan, W., Yang, M., Li, H., Wang, C., Wang, B.: End-to-end learning for high-precision lane keeping via multi-state model. CAAI Trans. Intell. Technol. 3(4), 185–190 (2018)

    Article  Google Scholar 

  28. Duong, M.-T., Do, T.-D., Le, M.-H.: Navigating self-driving vehicles using convolutional neural network. In: 2018 4th International Conference on Green Technology and Sustainable Development (GTSD), pp. 607–610. IEEE (2018)

  29. John, V., et al.: Estimation of steering angle and collision avoidance for automated driving using deep mixture of experts. IEEE Transactions on Intelligent Vehicles 3(4), 571–584 (2018)

    Article  Google Scholar 

  30. Chi, L., Mu, Y.: Deep steering: Learning end-to-end driving model from spatial and temporal visual cues. arXiv:1708.03798 (2017)

  31. Simmons, B., Adwani, P., Pham, H., Alhuthaifi, Y., Wolek, A.: Training a remote-control car to autonomously lane-follow using end-to-end neural networks. In: 2019 53rd Annual Conference on Information Sciences and Systems (CISS), pp. 1–6. IEEE (2019)

  32. Yang, Z., Zhang, Y., Yu, J., Cai, J., Luo, J.: End-to-end multi-modal multi-task vehicle control for self-driving cars with visual perceptions. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2289-2294. IEEE (2018)

  33. Xing, Y., Lv, C., Cao, D., Velenis, E.: Multi-scale driver behavior modeling based on deep spatial-temporal representation for intelligent vehicles. Transportation Research Part C: Emerging Technologies 130, 103288 (2021)

    Article  Google Scholar 

  34. Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proc. IEEE Conf. Comput. Vis.Pattern Recognit., pp. 4700–4708 (2017)

  35. Kim, J.-H., et al.: A study on an enhanced autonomous driving simulation model based on reinforcement learning using a collision prevention model. Electronics 10.18, 2271 (2021)

    Article  Google Scholar 

Download references

Acknowledgments

The work is support in part by the Ministry of science and Technology under grant MOST 110-2218-E-006-026 and 109-2221-E-150-004-MY3, Taiwan.

Funding

The work is support in part by the Ministry of science and Technology under grant MOST 110-2218-E-006-026 and 109-2221-E-150-004-MY3, Taiwan

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, National Cheng Kung University, No.1, University Road, Tainan city, 701, Taiwan

    Abida Khanum & Chu-Sing Yang

  2. Department of Aeronautical Engineering, National Formosa University, No.64, Wunhua Rd., Huwei Township, Yunlin Country, 632, Taiwan

    Chao-Yang Lee

  3. Institute of Data Science, National Cheng Kung University, No.1, University Road, Tainan city, 701, Taiwan

    Chih-Chung Hus

Authors
  1. Abida Khanum
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chao-Yang Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chih-Chung Hus
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chu-Sing Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Abida Khanum; Conceptualization, Methodology, writing-original draft preparation, Chao-Yang Lee; methodology, review, editing and funding, Chih-Chung Hus review and editing, Chu-Sing Yang; review and funding. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Chao-Yang Lee.

Ethics declarations

Ethics approval

No applicable

Consent for Publication

All authors have approved the manuscript and agree with its publication on Journal of Intelligent and Robotic Systems.

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Availability of data and materials

The data use to analysed during the study are available the corresponding authors on reasonable request.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Chao-Yang Lee, Chih-Chung Hus and Chu-Sing Yang are contributed equally to this work.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khanum, A., Lee, CY., Hus, CC. et al. Anticipating Autonomous Vehicle Driving based on Multi-Modal Multiple Motion Tasks Network. J Intell Robot Syst 105, 69 (2022). https://doi.org/10.1007/s10846-022-01677-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-022-01677-2

Keywords

Search

Navigation