Skip to main content
SpringerLink
Log in

DeepGuard: a framework for safeguarding autonomous driving systems from inconsistent behaviour

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

The deep neural networks (DNNs)-based autonomous driving systems (ADSs) are expected to reduce road accidents and improve safety in the transportation domain as it removes the factor of human error from driving tasks. The DNN-based ADS sometimes may exhibit erroneous or unexpected behaviours due to unexpected driving conditions which may cause accidents. Therefore, safety assurance is vital to the ADS. However, DNN-based ADS is a highly complex system that puts forward a strong demand for robustness, more specifically, the ability to predict unexpected driving conditions to prevent potential inconsistent behaviour. It is not possible to generalize the DNN model’s performance for all driving conditions. Therefore, the driving conditions that were not considered during the training of the ADS may lead to unpredictable consequences for the safety of autonomous vehicles. This study proposes an autoencoder and time series analysis–based anomaly detection system to prevent the safety-critical inconsistent behaviour of autonomous vehicles at runtime. Our approach called DeepGuard consists of two components. The first component- the inconsistent behaviour predictor, is based on an autoencoder and time series analysis to reconstruct the driving scenarios. Based on reconstruction error (e) and threshold (θ), it determines the normal and unexpected driving scenarios and predicts potential inconsistent behaviour. The second component provides on-the-fly safety guards, that is, it automatically activates healing strategies to prevent inconsistencies in the behaviour. We evaluated the performance of DeepGuard in predicting the injected anomalous driving scenarios using already available open-sourced DNN-based ADSs in the Udacity simulator. Our simulation results show that the best variant of DeepGuard can predict up to 93% on the CHAUFFEUR ADS, 83% on DAVE-2 ADS, and 80% of inconsistent behaviour on the EPOCH ADS model, outperforming SELFORACLE and DeepRoad. Overall, DeepGuard can prevent up to 89% of all predicted inconsistent behaviours of ADS by executing predefined safety guards.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. Deniosing autoencoder is denoted with capital letter “DAE” and deep autoencoder is denoted with small letter “dae”.

References

  • An, J., Cho, S.: Variational autoencoder based anomaly detection using reconstruction probability. Spec. Lect. IE. 2, 1–18 (2015)

    Google Scholar 

  • Anderson, J., Kalra, N., Stanley, K., Sorensen, P., Samaras, C., Oluwatola, O.: Autonomous vehicle technology: a guide for policymakers. RAND Corporation (2016)

  • Azzalini, D., Bonali, L., Amigoni, F.: A minimally supervised approach based on variational autoencoders for anomaly detection in autonomous robots. IEEE Robot. Autom. Lett. 6, 2985–2992 (2021). https://doi.org/10.1109/LRA.2021.3062597

    Article  Google Scholar 

  • Bahavan, N., Suman, N., Cader, S., Ranganayake, R., Seneviratne, D., Maddumage, V., Seneviratne, G., Supun, Y., Wijesiri, I., Dehigaspitiya, S., Tissera, D., Edussooriya, C.: Anomaly Detection using Deep Reconstruction and Forecasting for Autonomous Systems, (2020). arxiv.2006.14556

  • Bermad, N., Zemmoudj, S., Omar, M.: Securing vehicular platooning against vehicle platooning disruption (VPD) attacks. In: 2019 8th international conference on performance evaluation and modeling in wired and wireless networks (PEMWN). pp. 1–6. IEEE (2019)

  • Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., Zhang, J., Zhang, X., Zhao, J., Zieba, K.: End to End Learning for Self-Driving Cars (2016). arxiv.1604.07316

  • Bolte, J.A., Bar, A., Lipinski, D., Fingscheidt, T.: Towards corner case detection for autonomous driving. In: IEEE Intelligent Vehicles Symposium, Proceedings. pp. 438–445 (2019)

  • Borghesi, A., Bartolini, A., Lombardi, M., Milano, M., Benini, L.: A semisupervised autoencoder-based approach for anomaly detection in high performance computing systems. Eng. Appl. Artif. Intell. 85, 634–644 (2019). https://doi.org/10.1016/j.engappai.2019.07.008

    Article  Google Scholar 

  • Campos, G.O., Zimek, A., Sander, J., Campello, R.J.G.B., Micenková, B., Schubert, E., Assent, I., Houle, M.E.: On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study. Data Min. Knowl. Discov. 30, 891–927 (2016). https://doi.org/10.1007/s10618-015-0444-8

    Article  MathSciNet  Google Scholar 

  • Cerf, V.G.: A comprehensive self-driving car test. Commun. ACM. 61, 7–7 (2018). https://doi.org/10.1145/3177753

    Article  Google Scholar 

  • Chen, T., Liu, X., Xia, B., Wang, W., Lai, Y.: Unsupervised anomaly detection of industrial robots using sliding-window convolutional variational autoencoder. IEEE Access. (2020). https://doi.org/10.1109/ACCESS.2020.2977892

    Article  Google Scholar 

  • Chen, Z., Yeo, C.K., Lee, B.S., Lau, C.T.: Autoencoder-based network anomaly detection. In: 2018 Wireless Telecommunications Symposium (WTS). pp. 1–5. IEEE (2018)

  • Chong, Y.S., Tay, Y.H.: Abnormal event detection in videos using spatiotemporal autoencoder (2017)

  • Cozzolino, D., Verdoliva, L.: Single-image splicing localization through autoencoder-based anomaly detection. In: 2016 IEEE international workshop on information forensics and security (WIFS). pp. 1–6. IEEE (2016)

  • Davis, J., Goadrich, M.: The relationship between Precision-Recall and ROC curves. In: Proceedings of the 23rd international conference on Machine learning - ICML ’06. pp. 233–240. ACM Press, New York, New York, USA (2006)

  • Epoch Team: Steering Angle Model: Epoch, (2016) https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/cg23

  • Gomes, L.: Hidden Obstacles for Google’s Self-Driving Cars - MIT Technology Review, https://www.technologyreview.com/s/530276/hidden-obstacles-for-googles-self-driving-cars/

  • Greenblatt, N.A.: Self-driving cars and the law. IEEE Spectr. 53, 46–51 (2016). https://doi.org/10.1109/MSPEC.2016.7419800

    Article  Google Scholar 

  • Gyawali, S., Qian, Y., Hu, R.Q.: Machine learning and reputation based misbehavior detection in vehicular communication networks. IEEE Trans. Veh. Technol. 69, 8871–8885 (2020). https://doi.org/10.1109/TVT.2020.2996620

    Article  Google Scholar 

  • Haq, F.U., Shin, D., Nejati, S., Briand, L.C.: Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. In: Proceedings - 2020 IEEE 13th International conference on software testing, verification and validation, ICST 2020. pp. 85–95 (2020)

  • Henriksson, J., Berger, C., Borg, M., Tornberg, L., Englund, C., Sathyamoorthy, S.R., Ursing, S.: Towards structured evaluation of deep neural network supervisors. In: Proceedings - 2019 IEEE international conference on artificial intelligence testing, AITest 2019. pp. 27–34 (2019)

  • Hu, J., Xu, L., He, X., Meng, W.: Abnormal driving detection based on normalized driving behavior. IEEE Trans. Veh. Technol. 66, 6645–6652 (2017). https://doi.org/10.1109/TVT.2017.2660497

    Article  Google Scholar 

  • Jana, S., Tian, Y., Pei, K., Ray, B.: DeepTest: Automated testing of deep-neural-network-driven autonomous cars. In: Proceedings - international conference on software engineering (2018)

  • Kusano, K.D., Gabler, H.C.: Safety benefits of forward collision warning, brake assist, and autonomous braking systems in rear-end collisions. IEEE Trans. Intell. Transp. Syst. 13, 1546–1555 (2012). https://doi.org/10.1109/TITS.2012.2191542

    Article  Google Scholar 

  • Nguyen, V.-L., Lin, P.-C., Hwang, R.-H.: Enhancing misbehavior detection in 5G vehicle-to-vehicle communications. IEEE Trans. Veh. Technol. 69, 9417–9430 (2020). https://doi.org/10.1109/TVT.2020.2975822

    Article  Google Scholar 

  • NHTSA: Automated Vehicles for Safety (2020), https://www.nhtsa.gov/technology-innovation/automated-vehicles-safety

  • Patel, N., Nandini Saridena, A., Choromanska, A., Krishnamurthy, P., Khorrami, F.: Adversarial learning-based on-line anomaly monitoring for assured autonomy. In: IEEE international conference on intelligent robots and systems. pp. 6149–6154 (2018)

  • Pei, K., Cao, Y., Yang, J., Jana, S.: DeepXplore. In: Proceedings of the 26th Symposium on operating systems principles. pp. 1–18. ACM, New York, NY, USA (2017)

  • Saito, T., Rehmsmeier, M.: The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 10, e0118432 (2015). https://doi.org/10.1371/journal.pone.0118432

    Article  Google Scholar 

  • Stocco, A., Weiss, M., Calzana, M., Tonella, P.: Misbehaviour prediction for autonomous driving systems. In: Proceedings of the ACM/IEEE 42nd international conference on software engineering. pp. 359–371. ACM, New York, NY, USA (2020)

  • Team Chauffeur: self-driving-car/steering-models/community-models/chauffeur at master · udacity/self-driving-car · GitHub, (2016) https://github.com/udacity/self-driving-car/tree/master/steering-models/community-models/chauffeur

  • Tjonnas, J., Johansen, T.A.: Stabilization of automotive vehicles using active steering and adaptive brake control allocation. IEEE Trans. Control Syst. Technol. 18, 545–558 (2010). https://doi.org/10.1109/TCST.2009.2023981

    Article  Google Scholar 

  • Udacity: a self-driving car simulator built with Unity (2017), https://github.com/udacity/self-driving-car-sim

  • Unity: Unity - Scripting API: ParticleSystem, https://docs.unity3d.com/ScriptReference/ParticleSystem.html

  • Unity technologies: unity real-time development platform | 3D, 2D VR & AR Engine, (2019) https://unity.com/

  • Wang, X., Liu, J., Qiu, T., Mu, C., Chen, C., Zhou, P.: A real-time collision prediction mechanism with deep learning for intelligent transportation system. IEEE Trans. Veh. Technol. 69, 9497–9508 (2020). https://doi.org/10.1109/TVT.2020.3003933

    Article  Google Scholar 

  • World Health Organization: Global Status Report on Road Safety 2018, https://www.who.int/publications/i/item/9789241565684

  • Wu, M., Zeng, H., Wang, C., Yu, H.: Safety Guard. In: Proceedings of the 54th annual design automation conference 2017. pp. 1–6. ACM, New York, NY, USA (2017)

  • Xu, H., Feng, Y., Chen, J., Wang, Z., Qiao, H., Chen, W., Zhao, N., Li, Z., Bu, J., Li, Z., Liu, Y., Zhao, Y., Pei, D.: Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in Web Applications. In: Proceedings of the 2018 world wide web conference on world wide web - WWW ’18. pp. 187–196. ACM Press, New York, New York, USA (2018)

  • Zavrtanik, V., Kristan, M., Skočaj, D.: Reconstruction by inpainting for visual anomaly detection. Pattern Recognit. (2021). https://doi.org/10.1016/j.patcog.2020.107706

    Article  Google Scholar 

  • Zhang, M., Zhang, Y., Zhang, L., Liu, C., Khurshid, S.: DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering. pp. 132–142. ACM, New York, NY, USA (2018)

  • Zhang, W., Wang, Z., Drugge, L., Nybacka, M.: Evaluating model predictive path following and yaw stability controllers for over-actuated autonomous electric vehicles. IEEE Trans. Veh. Technol. 69, 12807–12821 (2020). https://doi.org/10.1109/TVT.2020.3030863

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by the Korea National Research Foundation of the Ministry of ICT and Science (NRF2020R1A2C1007571).

Author information

Authors and Affiliations

  1. School of Electrical and Computer Engineering, Chungbuk National University, Chung-daero, Seowon-gu, Cheongju-si, Chungbuk, South Korea

    Manzoor Hussain, Nazakat Ali & Jang-Eui Hong

Authors
  1. Manzoor Hussain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nazakat Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jang-Eui Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jang-Eui Hong.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hussain, M., Ali, N. & Hong, JE. DeepGuard: a framework for safeguarding autonomous driving systems from inconsistent behaviour. Autom Softw Eng 29, 1 (2022). https://doi.org/10.1007/s10515-021-00310-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10515-021-00310-0

Keywords

Search

Navigation