Skip to main content
SpringerLink
Log in

Formally Verified Safety Net for Waypoint Navigation Neural Network Controllers

  • Conference paper
  • First Online:
Formal Methods (FM 2021)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 13047))

Included in the following conference series:

Abstract

This paper describes a formal model of a “location, heading and speed” waypoint navigation task for an autonomous ground vehicle—that is, a task of navigating the vehicle towards a particular location so that it has the desired heading and speed when in that location. Our novel way of modeling this task makes formal reasoning over controller correctness tractable. We state our model in differential dynamic logic (dL), which we then use to establish a formal definition of waypoint feasibility and formally verify its validity in the KeYmaera X interactive theorem prover. The formal machine-checked proof witnesses that for any waypoint we consider feasible, the vehicle can indeed be controlled to reach it within the prescribed error bound. We also describe how we use these formal definitions and theorem statements to inform training of neural network controllers for performing this waypoint navigation task. Note that in our approach we do not need to rely on the neural network controller always being perfect—instead, the formal model allows a synthesis of a correct-by-construction safety net for the controller that checks whether the neural network output is safe to act upon and present a safe alternative if it is not.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    As a consequence of instantaneous steering, the curvature of the vehicle’s path is able to change instantaneously to any value in the feasible range (that is, between \(-\frac{1}{R_\texttt {min}}\) and \(\frac{1}{R_\texttt {min}}\), where \(R_\texttt {min}\) is the minimum turning radius of the vehicle).

  2. 2.

    Some solvers, e.g. dReal [14], opt for \(\delta \)-decidability to render transcendental functions decidable.

References

  1. Achiam, J., Held, D., Tamar, A., Abbeel, P.: Constrained policy optimization. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, pp. 22–31 (2017). proceedings.mlr.press/v70/achiam17a.html

  2. Ahn, E.: Towards Safe Reinforcement Learning in the Real World. Master’s thesis, Carnegie Mellon University, cMU-RI-TR-19-56 (2019). www.ri.cmu.edu/wp-content/uploads/2019/08/MSR/Thesis/Edward/Ahn/2019.pdf

  3. Alonso-Mora, J., DeCastro, J.A., Raman, V., Rus, D., Kress-Gazit, H.: Reactive mission and motion planning with deadlock resolution avoiding dynamic obstacles. Auton. Robot. 42(4), 801–824 (2017). https://doi.org/10.1007/s10514-017-9665-6

  4. Bohrer, B., Rahli, V., Vukotic, I., Völp, M., Platzer, A.: Formally verified differential dynamic logic. In: Proceedings of the 6th ACM SIGPLAN Conference on Certified Programs and Proofs. ACM (2017). https://doi.org/10.1145/3018610.3018616

  5. Bohrer, B., Tan, Y.K., Mitsch, S., Sogokon, A., Platzer, A.: A formal safety net for waypoint following in ground robots. IEEE Robot. Automat. Lett. 4(3), 2910–2917 (2019). https://doi.org/10.1109/LRA.2019.2923099

  6. Chang, Y.C., Gao, S.: Stabilizing neural control using self-learned almost Lyapunov critics. In: Proceedings of the 2021 International Conference on Robotics and Automation (ICRA 2021) (2021). arxiv.org/abs/2107.04989

  7. Chang, Y.C., Roohi, N., Gao, S.: Neural Lyapunov control (2020). arxiv.org/abs/2005.00611

  8. Davenport, J.H., Heintz, J.: Real quantifier elimination is doubly exponential. J. Symb. Comput. 5(1/2), 29–35 (1988). https://doi.org/10.1016/S0747-7171(88)80004-X

    Article  MathSciNet  MATH  Google Scholar 

  9. Desai, A., Saha, I., Yang, J., Qadeer, S., Seshia, S.A.: DRONA: a framework for safe distributed mobile robotics. In: Martínez, S., Tovar, E., Gill, C., Sinopoli, B. (eds.) Proceedings of the 8th International Conference on Cyber-Physical Systems, ICCPS 2017, Pittsburgh, Pennsylvania, USA, pp. 239–248. ACM (2017). https://doi.org/10.1145/3055004.3055022

  10. Dreossi, T., Donzé, A., Seshia, S.A.: Compositional falsification of cyber-physical systems with machine learning Components. J. Autom. Reason. 63(4), 1031–1053 (2019). https://doi.org/10.1007/s10817-018-09509-5

  11. Fisher, M., Mascardi, V., Rozier, K.Y., Schlingloff, B.-H., Winikoff, M., Yorke-Smith, N.: Towards a framework for certification of reliable autonomous systems. Auton. Agents Multi-Agent Syst. 35(1), 1–65 (2020). https://doi.org/10.1007/s10458-020-09487-2

  12. Foughali, M., Bensalem, S., Combaz, J., Ingrand, F.: Runtime verification of timed properties in autonomous robots. In: 18th ACM/IEEE International Conference on Formal Methods and Models for System Design, MEMOCODE 2020, Jaipur, India, pp. 1–12. IEEE (2020). https://doi.org/10.1109/MEMOCODE51338.2020.9315156

  13. Fulton, N., Mitsch, S., Quesel, J., Völp, M., Platzer, A.: Keymaera X: an axiomatic tactical theorem prover for hybrid systems. In: Felty, A.P., Middeldorp, A. (eds.) Automated Deduction - CADE-25 - 25th International Conference on Automated Deduction, Berlin, Germany, 2015, Proceedings. Lecture Notes in Computer Science, vol. 9195, pp. 527–538. Springer (2015). https://doi.org/10.1007/978-3-319-21401-6_36

  14. Gao, S., Kong, S., Clarke, E.M.: dreal: An SMT solver for nonlinear theories over the reals. In: Bonacina, M.P. (ed.) Automated Deduction - CADE-24 - 24th International Conference on Automated Deduction, Lake Placid, NY, USA, 2013. Proceedings. Lecture Notes in Computer Science, vol. 7898, pp. 208–214. Springer (2013). https://doi.org/10.1007/978-3-642-38574-2_14

  15. Hoxha, B., Fainekos, G.E.: Planning in dynamic environments through temporal logic monitoring. In: Magazzeni, D., Sanner, S., Thiébaux, S. (eds.) Planning for Hybrid Systems, Papers from the 2016 AAAI Workshop, Phoenix, Arizona, USA 2016. AAAI Workshops, vol. WS-16-12. AAAI Press (2016). www.aaai.org/ocs/index.php/WS/AAAIW16/paper/view/12556

  16. Huang, J., et al.: ROSRV: runtime verification for robots. In: Bonakdarpour, B., Smolka, S.A. (eds.) RV 2014. LNCS, vol. 8734, pp. 247–254. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11164-3_20

    Chapter  Google Scholar 

  17. Ivanov, R., Carpenter, T.J., Weimer, J., Alur, R., Pappas, G.J., Lee, I.: Verifying the safety of autonomous systems with neural network controllers. ACM Trans. Embed. Comput. Syst. 20(1), 1–26 (2021). https://doi.org/10.1145/3419742

  18. Kania, R., Frederick, P., Pritchett, W., Wood, B., Mentzer, C., Johnson, E.: Dismounted soldier autonomy tools (DSAT) – from conception to deployment. In: 2014 NDIA Ground Vehicles Systems Engineering and Technology Symposium (2014). gvsets.ndia-mich.org/publication.php?documentID=171

  19. Kempa, B., Zhang, P., Jones, P.H., Zambreno, J., Rozier, K.Y.: Embedding online runtime verification for fault disambiguation on robonaut2. In: Bertrand, N., Jansen, N. (eds.) Formal Modeling and Analysis of Timed Systems - 18th International Conference, FORMATS 2020, Vienna, Austria, Proceedings. Lecture Notes in Computer Science, vol. 12288, pp. 196–214. Springer (2020). https://doi.org/10.1007/978-3-030-57628-8_12

  20. Liu, S.B., Roehm, H., Heinzemann, C., Lütkebohle, I., Oehlerking, J., Althoff, M.: Provably safe motion of mobile robots in human environments. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2017, Vancouver, BC, Canada, pp. 1351–1357. IEEE (2017). https://doi.org/10.1109/IROS.2017.8202313

  21. Luckcuck, M., Farrell, M., Dennis, L.A., Dixon, C., Fisher, M.: Formal specification and verification of autonomous robotic systems: a survey. ACM Comput. Surv. 52(5), 1–41 (2019). https://doi.org/10.1145/3342355

  22. Milner, R.: LCF: a way of doing proofs with a machine. In: Becvár, J. (ed.) Mathematical Foundations of Computer Science 1979, Proceedings, 8th Symposium Lecture Notes in Computer Science. Olomouc, Czechoslovakia, vol. 74, pp. 146–159. Springer (1979). https://doi.org/10.1007/3-540-09526-8_11

  23. Mitsch, S., Ghorbal, K., Vogelbacher, D., Platzer, A.: Formal verification of obstacle avoidance and navigation of ground robots. I. J. Robotics Res. 36(12), 1312–1340 (2017). https://doi.org/10.1177/0278364917733549

    Article  Google Scholar 

  24. Mitsch, S., Platzer, A.: ModelPlex: verified runtime validation of verified cyber-physical system models. Formal Methods Syst. Design (1), 33–74 (2016). https://doi.org/10.1007/s10703-016-0241-z

  25. Pan, Y., Lin, Q., Shah, H., Dolan, J.M.: Safe planning for self-driving via adaptive constrained ILQR. CoRR abs/2003.02757 (2020). arxiv.org/abs/2003.02757

  26. Platzer, A.: Differential dynamic logic for hybrid systems. J. Autom. Reasoning 41(2), 143–189 (2008). https://doi.org/10.1007/s10817-008-9103-8

    Article  MathSciNet  MATH  Google Scholar 

  27. Platzer, A.: A Complete Uniform Substitution Calculus for Differential Dynamic Logic. J. Autom. Reason. 59(2), 219–265 (2016). https://doi.org/10.1007/s10817-016-9385-1

  28. Platzer, A.: Logical Foundations of Cyber-Physical Systems. Springer (2018). https://doi.org/10.1007/978-3-319-63588-0

    Article  Google Scholar 

  29. Platzer, A., Clarke, E.M.: Formal verification of curved flight collision avoidance maneuvers: a case study. In: Cavalcanti, A., Dams, D. (eds.) FM. LNCS, vol. 5850, pp. 547–562. Springer (2009). https://doi.org/10.1007/978-3-642-05089-3_35

  30. Platzer, A., Tan, Y.K.: Differential equation invariance axiomatization. J. ACM 67(1), 1–66 (2020). https://doi.org/10.1145/3380825

  31. Rizaldi, A., Immler, F., Schürmann, B., Althoff, M.: A formally verified motion planner for autonomous vehicles. In: Lahiri, S.K., Wang, C. (eds.) Automated Technology for Verification and Analysis - 16th International Symposium, ATVA 2018, Los Angeles, CA, USA, 2018, Proceedings. Lecture Notes in Computer Science, vol. 11138, pp. 75–90. Springer (2018). https://doi.org/10.1007/978-3-030-01090-4_5

  32. Schulman, J., Levine, S., Moritz, P., Jordan, M.I., Abbeel, P.: Trust region policy optimization. CoRR abs/1502.05477 (2015). arxiv.org/abs/1502.05477

  33. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms (2017). arxiv.org/abs/1707.06347v2

  34. Seegmiller, N.: Dynamic model formulation and calibration for wheeled mobile robots. Ph.D. thesis, Carnegie Mellon University, Pittsburgh, PA (2014). www.ri.cmu.edu/publications/dynamic-model-formulation-and-calibration-for-wheeled-mobile-robots/

  35. Seegmiller, N., Kelly, A.: High-fidelity yet fast dynamic models of wheeled mobile robots. IEEE Trans. Robot. 32(3), 614–625 (2016). https://doi.org/10.1109/TRO.2016.2546310

  36. Shivakumar, S., Torfah, H., Desai, A., Seshia, S.A.: SOTER on ROS: a run-time assurance framework on the robot operating system. In: Deshmukh, J., Ničković, D. (eds.) RV 2020. LNCS, vol. 12399, pp. 184–194. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60508-7_10

    Chapter  Google Scholar 

  37. Tran, H., et al.: NNV: the neural network verification tool for deep neural networks and learning-enabled cyber-physical systems. In: Lahiri, S.K., Wang, C. (eds.) Computer Aided Verification - 32nd International Conference, CAV 2020, Los Angeles, CA, USA, 2020, Proceedings, Part I. Lecture Notes in Computer Science, vol. 12224, pp. 3–17. Springer (2020). https://doi.org/10.1007/978-3-030-53288-8_1

  38. Tuncali, C.E., Fainekos, G., Prokhorov, D.V., Ito, H., Kapinski, J.: Requirements-driven test generation for autonomous vehicles with machine learning components. IEEE Trans. Intell. Veh. 5(2), 265–280 (2020). https://doi.org/10.1109/TIV.2019.2955903

    Article  Google Scholar 

  39. Tuncali, C.E., Kapinski, J., Ito, H., Deshmukh, J.V.: Reasoning about safety of learning-enabled components in autonomous cyber-physical systems. In: Proceedings of the 55th Annual Design Automation Conference, DAC 2018, San Francisco, CA, USA, pp. 1–6. ACM (2018). https://doi.org/10.1145/3195970.3199852

  40. Wong, K.W., Ehlers, R., Kress-Gazit, H.: Resilient, provably-correct, and high-level robot behaviors. IEEE Trans. Robot. 34(4), 936–952 (2018). https://doi.org/10.1109/TRO.2018.2830353

  41. Wong, K.W., Finucane, C., Kress-Gazit, H.: Provably-correct robot control with ltlmop, OMPL and ROS. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, p. 2073. IEEE (2013). https://doi.org/10.1109/IROS.2013.6696636

Download references

Acknowledgment

This material is based upon work supported by the United States Air Force and DARPA under Contract No. FA8750-18-C-0092. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of the United States Air Force and DARPA. Distribution Statement “A” (Approved for Public Release, Distribution Unlimited).

Author information

Authors and Affiliations

  1. HRL Laboratories, LLC, Malibu, CA, USA

    Alexei Kopylov, Aleksey Nogin & Michael Warren

  2. Carnegie Mellon University, Pittsburgh, PA, USA

    Stefan Mitsch

Authors
  1. Alexei Kopylov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefan Mitsch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aleksey Nogin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Warren
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aleksey Nogin .

Editor information

Editors and Affiliations

  1. University of Twente, Enschede, The Netherlands

    Marieke Huisman

  2. NASA Ames Research Center, Moffett Field, CA, USA

    Corina Păsăreanu

  3. Institute of Software, Chinese Academy of Sciences, Beijing, China

    Naijun Zhan

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kopylov, A., Mitsch, S., Nogin, A., Warren, M. (2021). Formally Verified Safety Net for Waypoint Navigation Neural Network Controllers. In: Huisman, M., Păsăreanu, C., Zhan, N. (eds) Formal Methods. FM 2021. Lecture Notes in Computer Science(), vol 13047. Springer, Cham. https://doi.org/10.1007/978-3-030-90870-6_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-90870-6_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-90869-0

  • Online ISBN: 978-3-030-90870-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation