Skip to main content
SpringerLink
Log in

Formal Assurance for Cooperative Intelligent Autonomous Agents

  • Conference paper
  • First Online:
NASA Formal Methods (NFM 2018)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 10811))

Included in the following conference series:

Abstract

Developing trust in intelligent agents requires understanding the full capabilities of the agent, including the boundaries beyond which the agent is not designed to operate. This paper focuses on applying formal verification methods to identify these boundary conditions in order to ensure the proper design for the effective operation of the human-agent team. The approach involves creating an executable specification of the human-machine interaction in a cognitive architecture, which incorporates the expression of learning behavior. The model is then translated into a formal language, where verification and validation activities can occur in an automated fashion. We illustrate our approach through the design of an intelligent copilot that teams with a human in a takeoff operation, while a contingency scenario involving an engine-out is potentially executed. The formal verification and counterexample generation enables increased confidence in the designed procedures and behavior of the intelligent copilot system.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Newell, A., Shaw, J.C., Simon, H.A.: Report on a general problem-solving program. In: Proceedings of the International Conference on Information Processing, pp. 256–264 (1959)

    Google Scholar 

  2. Buchanan, B.G., Shortliffe, E.H.: Rule Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project. The Addison-Wesley Series in Artificial Intelligence. Addison-Wesley Longman Publishing Co., Inc., Boston (1984)

    Google Scholar 

  3. Anderson, J.R., Matessa, M., Lebiere, C.: ACT-R: a theory of higher level cognition and its relation to visual attention. Hum.-Comput. Interact. 12(4), 439–462 (1997)

    Article  Google Scholar 

  4. Laird, J.E.: The SOAR Cognitive Architecture. MIT Press, Cambridge (2012)

    Google Scholar 

  5. Sutton, R.L., Barto, B.: Reinforcement Learning. MIT Press, Cambridge (2008)

    Google Scholar 

  6. Mittal, S., Douglass, S.A.: Net-centric ACT-R based cognitive architecture with DEVS unified process. In: DEVS Symposium Spring Simulation Multiconference, Boston (2011)

    Google Scholar 

  7. Garlan, D., Cheng, S., Huang, A., Schmerl, B., Steenkiste, P.: Rainbow: architecture-based self adaptation with reusable infrastructure. Computer 37(10), 46–54 (2004)

    Article  Google Scholar 

  8. Wen, M., Ehlers, R., Topcu, U.: Correct-by-synthesis reinforcement learning with temporal logic constraints. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (2015)

    Google Scholar 

  9. Sharifloo, A.M., Spoletini, P.: LOVER: Light-weight fOrmal Verification of adaptivE systems at Run time. In: Păsăreanu, C.S., Salaün, G. (eds.) FACS 2012. LNCS, vol. 7684, pp. 170–187. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-35861-6_11

    Chapter  Google Scholar 

  10. Curzon, P., Ruknas, R., Blandford, A.: An approach to formal verification of human computer interaction. Form. Asp. Comput. 19, 513–550 (2007)

    Article  MATH  Google Scholar 

  11. O’Conner, M., Tangirala, S., Kumar, R., Bhattacharyya, S., Sznaier, S., Holloway, L.: A bottom-up approach to verification of hybrid model-based hierarchical controllers with application to underwater vehicles. In: Proceedings of American Control Conference (2006)

    Google Scholar 

  12. Rocha, C., Cadavid, H., Muñoz, C., Siminiceanu, R.: A formal interactive verification environment for the plan execution interchange language. In: Derrick, J., Gnesi, S., Latella, D., Treharne, H. (eds.) IFM 2012. LNCS, vol. 7321, pp. 343–357. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30729-4_24

    Chapter  Google Scholar 

  13. Dowek, G., Munoz, C., Pasareanu, C.: A small-step semantics of PLEXIL (2008)

    Google Scholar 

  14. Dowek, G., Munoz, C., Pasareanu, C.: A formal analysis framework for PLEXIL. In: Proceedings of 3rd Workshop on Planning and Plan Execution for Real-World Systems (2007)

    Google Scholar 

  15. Dowek, G., Munoz, C., Rocha, C.: Rewriting logic semantics of a plan execution language. In: EPTCS, vol. 18, pp. 77–91 (2009)

    Google Scholar 

  16. Strauss, P.J.: Executable semantics for PLEXIL: simulating a task-scheduling language in Haskell. Masters thesis (2009)

    Google Scholar 

  17. Balasubramanian, D., Pasareanu, C., Whalen, M.W., Karsai, G., Lowry, M.R.: Polyglot: modeling and analysis for multiple statechart formalisms. In: Dwyer, M.B., Tip, F. (eds.) ISSTA. ACM (2011)

    Google Scholar 

  18. Verdejo, A., Martí-Oliet, N.: Two case studies of semantics execution in Maude: CCS and LOTOS. Formal Methods Syst. Des. 27, 113–172 (2005)

    Article  MATH  Google Scholar 

  19. Eskridge, T.C., Carvalho, M.M., Bhattacharyya, S., Vogl, T.: Verifiable autonomy final report. Technical report, Florida Institute of Technology and Rockwell Collins (2015)

    Google Scholar 

  20. Cimatti, A., Clarke, E., Giunchiglia, E., Giunchiglia, F., Pistore, M., Roveri, M., Sebastiani, R., Tacchella, A.: NuSMV 2: an opensource tool for symbolic model checking. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 359–364. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45657-0_29

    Chapter  Google Scholar 

  21. Uppaal website (2010). http://www.uppaal.org

  22. Owre, S., Rajan, S., Rushby, J.M., Shankar, N., Srivas, M.: PVS: combining specification, proof checking, and model checking. In: Alur, R., Henzinger, T.A. (eds.) CAV 1996. LNCS, vol. 1102, pp. 411–414. Springer, Heidelberg (1996). https://doi.org/10.1007/3-540-61474-5_91

    Chapter  Google Scholar 

  23. de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78800-3_24

    Chapter  Google Scholar 

  24. Larsen, K.G., Pettersson, P., Yi, W.: Model-checking for real-time systems. In: Reichel, H. (ed.) FCT 1995. LNCS, vol. 965, pp. 62–88. Springer, Heidelberg (1995). https://doi.org/10.1007/3-540-60249-6_41

    Chapter  Google Scholar 

  25. Bengtsson, J., Larsen, K., Larsson, F., Pettersson, P., Yi, W.: UPPAAL: a tool suite for automatic verification of real-time systems. Theor. Comput. Sci. (1996). RS-96-58

    Google Scholar 

  26. Alur, R., David, L.D.: A theory of timed automata. Theor. Comput. Sci. 126, 183–235 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  27. Bozga, M., Daws, C., Maler, O., Olivero, A., Tripakis, S., Yovine, S.: Kronos: a model-checking tool for real-time systems. In: Ravn, A.P., Rischel, H. (eds.) FTRTFT 1998. LNCS, vol. 1486, pp. 298–302. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0055357

    Chapter  Google Scholar 

  28. Neogi, N.A.: Capturing safety requirements to enable effective task allocation between humans and automaton in increasingly autonomous systems. In: Proceedings of the AIAA Aviation Forum. 16th AIAA Aviation Technology, Integration, and Operations Conference (AIAA 2016-3594) (2016)

    Google Scholar 

  29. Code of Federal Regulations: Title 14 Aeronautics and Space. Federal Register, May 1962. http://www.ecfr.gov/cgi-bin/text

  30. The Boeing Company: Boeing 737 pilots operating handbook. Continental Airlines, November 2002. http://air.felisnox.com/view.php?name=737.pdf

  31. Official x-plane website (2016). http://www.x-plane.com

Download references

Author information

Authors and Affiliations

  1. School of Computing, Florida Institute of Technology, Melbourne, FL, USA

    Siddhartha Bhattacharyya, Thomas C. Eskridge, Marco Carvalho & Milton Stafford

  2. NASA Langley Research Center, Hampton, VA, USA

    Natasha A. Neogi

Authors
  1. Siddhartha Bhattacharyya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas C. Eskridge
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Natasha A. Neogi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marco Carvalho
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Milton Stafford
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Siddhartha Bhattacharyya .

Editor information

Editors and Affiliations

  1. NASA Langley Research Center, Hampton, Virginia, USA

    Aaron Dutle

  2. NASA Langley Research Center, Hampton, Virginia, USA

    César Muñoz

  3. NASA Langley Research Center, Hampton, Virginia, USA

    Anthony Narkawicz

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bhattacharyya, S., Eskridge, T.C., Neogi, N.A., Carvalho, M., Stafford, M. (2018). Formal Assurance for Cooperative Intelligent Autonomous Agents. In: Dutle, A., Muñoz, C., Narkawicz, A. (eds) NASA Formal Methods. NFM 2018. Lecture Notes in Computer Science(), vol 10811. Springer, Cham. https://doi.org/10.1007/978-3-319-77935-5_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-77935-5_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-77934-8

  • Online ISBN: 978-3-319-77935-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation