Skip to main content
SpringerLink
Log in

Reasoning over Test Specifications Using Assume-Guarantee Contracts

  • Conference paper
  • First Online:
NASA Formal Methods (NFM 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13903))

Included in the following conference series:

  • 516 Accesses

Abstract

We establish a framework to reason about test campaigns described formally. First, we introduce the notion of a test structure—an object that carries i) the formal specifications of the system under test, and ii) the test objective, which is specified by a test engineer. We build on test structures to define test campaigns and specifications for the tester. Secondly, we use the algebra of assume-guarantee contracts to reason about constructing tester specifications, comparing test structures and test campaigns, and combining and splitting test structures. Using the composition operator, we characterize the conditions on the constituent tester specifications and test objectives for feasibly combining test structures. We illustrate the different applications of the quotient operator to split the test objective, the system into subsystems, or both. Finally, we illustrate test executions corresponding to the combined and split test structures in a discrete autonomous driving example and an aircraft formation-flying example. We anticipate that reasoning over test specifications would aid in generating optimal test campaigns.

A. Badithela, J. B. Graebener and I. Incer—These authors contributed equally to this work.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Atkinson, C., Groß, H.G.: Built-in contract testing in model-driven, component-based development. In: Proceedings of ICSR-7 Workshop on Component-Based Development Processes (2002)

    Google Scholar 

  2. Badithela, A., Graebener, J.B., Ubellacker, W., Mazumdar, E.V., Ames, A.D., Murray, R.M.: Synthesizing reactive test environments for autonomous systems: testing reach-avoid specifications with multi-commodity flows. In: 2023 International Conference on Robotics and Automation (ICRA), London, UK. IEEE, arXiv preprint arXiv:2210.10304 (2023, to appear)

  3. Baier, C., Katoen, J.P.: Principles of Model Checking. MIT Press, Cambridge (2008)

    MATH  Google Scholar 

  4. Beer, A., Ramler, R.: The role of experience in software testing practice. In: 2008 34th Euromicro Conference Software Engineering and Advanced Applications. pp. 258–265. IEEE (2008)

    Google Scholar 

  5. Benveniste, A., Caillaud, B., Ferrari, A., Mangeruca, L., Passerone, R., Sofronis, C.: Multiple viewpoint contract-based specification and design. In: de Boer, F.S., Bonsangue, M.M., Graf, S., de Roever, W.-P. (eds.) FMCO 2007. LNCS, vol. 5382, pp. 200–225. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-92188-2_9

    Chapter  MATH  Google Scholar 

  6. Benveniste, A., et al.: Contracts for system design. Found. Trends® Electron. Design Autom. 12(2–3), 124–400 (2018)

    Google Scholar 

  7. Blundell, C., Giannakopoulou, D., Pundefinedsundefinedreanu, C.S.: Assume-guarantee testing, SAVCBS 2005, p. 1-es. Association for Computing Machinery, New York (2005). https://doi.org/10.1145/1123058.1123060

  8. Brandán Briones, L.: Assume-guarantee reasoning with ioco testing relation. On testing software and systems: short papers, p. 103 (2010)

    Google Scholar 

  9. Brat, G., Jonsson, A.: Challenges in verification and validation of autonomous systems for space exploration. In: Proceedings of 2005 IEEE International Joint Conference on Neural Networks, vol. 5, pp. 2909–2914. IEEE (2005)

    Google Scholar 

  10. Bruno, M., Canfora, G., Di Penta, M., Esposito, G., Mazza, V.: Using test cases as contract to ensure service compliance across releases. In: Benatallah, B., Casati, F., Traverso, P. (eds.) ICSOC 2005. LNCS, vol. 3826, pp. 87–100. Springer, Heidelberg (2005). https://doi.org/10.1007/11596141_8

    Chapter  Google Scholar 

  11. Cofer, D., Gacek, A., Miller, S., Whalen, M.W., LaValley, B., Sha, L.: Compositional verification of architectural models. In: Goodloe, A.E., Person, S. (eds.) NFM 2012. LNCS, vol. 7226, pp. 126–140. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28891-3_13

    Chapter  Google Scholar 

  12. Dahm, W.J.: Technology horizons vision for the air force during 2010–2030 (video). Technical report, Chief Scientist (Air Force) Washington, DC (2011)

    Google Scholar 

  13. Dai, G., Bai, X., Wang, Y., Dai, F.: Contract-based testing for web services. In: 31st Annual International Computer Software and Applications Conference (COMPSAC 2007), vol. 1, pp. 517–526 (2007). https://doi.org/10.1109/COMPSAC.2007.100

  14. Dijkstra, E.W.: Guarded commands, nondeterminacy and formal derivation of programs. Commun. ACM 18(8), 453–457 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  15. Fremont, D.J., et al.: Formal scenario-based testing of autonomous vehicles: from simulation to the real world. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1–8. IEEE (2020)

    Google Scholar 

  16. Giannakopoulou, D., Păsăreanu, C., Blundell, C.: Assume-guarantee testing for software components. IET Softw. 2(6), 547–562 (2008)

    Article  Google Scholar 

  17. Graebener, J.B., Badithela, A., Murray, R.M.: Towards better test coverage: Merging unit tests for autonomous systems. In: Deshmukh, J.V., Havelund, K., Perez, I. (eds.) NFM 2022. LNCS, vol. 13260, pp. 133–155. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-06773-0_7

    Chapter  Google Scholar 

  18. Heckel, R., Lohmann, M.: Towards contract-based testing of web services. Electron. Notes Theor. Comput. Sci. 116, 145–156 (2005). https://doi.org/10.1016/j.entcs.2004.02.073, https://www.sciencedirect.com/science/article/pii/S1571066104052831. Proceedings of the International Workshop on Test and Analysis of Component Based Systems (TACoS 2004)

  19. Helle, P., Schamai, W., Strobel, C.: Testing of autonomous systems-challenges and current state-of-the-art. In: INCOSE International Symposium, vol. 26, pp. 571–584. Wiley Online Library (2016)

    Google Scholar 

  20. Incer, I., Sangiovanni-Vincentelli, A.L., Lin, C.W., Kang, E.: Quotient for assume-guarantee contracts. In: 16th ACM-IEEE International Conference on Formal Methods and Models for System Design, MEMOCODE 2018, pp. 67–77 (2018). https://doi.org/10.1109/MEMCOD.2018.8556872

  21. Incer, I.: The algebra of contracts. Ph.D. thesis, EECS Department, University of California, Berkeley (2022)

    Google Scholar 

  22. Incer, I., Mangeruca, L., Villa, T., Sangiovanni-Vincentelli, A.: The quotient in preorder theories. arXiv:2009.10886 (2020)

  23. Lamport, L.: Win and sin: predicate transformers for concurrency. ACM Trans. Program. Lang. Syst. (TOPLAS) 12(3), 396–428 (1990)

    Article  Google Scholar 

  24. Meyer, B.: Applying ‘design by contract’. Computer 25(10), 40–51 (1992)

    Article  Google Scholar 

  25. Negulescu, R.: Process Spaces. In: Palamidessi, C. (ed.) CONCUR 2000. LNCS, vol. 1877, pp. 199–213. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-44618-4_16

    Chapter  Google Scholar 

  26. Nuzzo, P., Sangiovanni-Vincentelli, A.L., Bresolin, D., Geretti, L., Villa, T.: A platform-based design methodology with contracts and related tools for the design of cyber-physical systems. Proc. IEEE 103(11), 2104–2132 (2015)

    Article  Google Scholar 

  27. Passerone, R., Incer, I., Sangiovanni-Vincentelli, A.L.: Coherent extension, composition, and merging operators in contract models for system design. ACM Trans. Embed. Comput. Syst. (TECS) 18(5s), 1–23 (2019)

    Article  Google Scholar 

  28. Pnueli, A.: The temporal logic of programs. In: 18th Annual Symposium on Foundations of Computer Science (SFCS 1977), pp. 46–57. IEEE (1977)

    Google Scholar 

  29. Sangiovanni-Vincentelli, A.L., Damm, W., Passerone, R.: Taming Dr. Frankenstein: contract-based design for cyber-physical systems. Eur. J. Control 18(3), 217–238 (2012). https://doi.org/10.3166/ejc.18.217-238

    Article  MathSciNet  MATH  Google Scholar 

  30. Seshia, S.A., Sadigh, D., Sastry, S.S.: Towards verified artificial intelligence. arXiv preprint arXiv:1606.08514 (2016)

  31. Tretmans, J.: Model based testing with labelled transition systems. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing. LNCS, vol. 4949, pp. 1–38. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78917-8_1

    Chapter  Google Scholar 

  32. Weiss, L.G.: Autonomous robots in the fog of war. IEEE Spectr. 48(8), 30–57 (2011)

    Article  Google Scholar 

Download references

Acknowledgements

The authors acknowledge funding from AFOSR Test and Evaluation program, grant FA9550-19-1-0302, and NSF and ASEE through an eFellows postdoctoral fellowship. The contents are solely the responsibility of the authors and do not necessarily represent the views of the sponsors.

Author information

Authors and Affiliations

  1. California Institute of Technology, Pasadena, USA

    Apurva Badithela, Josefine B. Graebener, Inigo Incer & Richard M. Murray

  2. University of California, Berkeley, Berkeley, USA

    Inigo Incer

Authors
  1. Apurva Badithela
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Josefine B. Graebener
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Inigo Incer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard M. Murray
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Josefine B. Graebener .

Editor information

Editors and Affiliations

  1. Iowa State University, Ames, IA, USA

    Kristin Yvonne Rozier

  2. University of Texas at Austin, Austin, TX, USA

    Swarat Chaudhuri

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Badithela, A., Graebener, J.B., Incer, I., Murray, R.M. (2023). Reasoning over Test Specifications Using Assume-Guarantee Contracts. In: Rozier, K.Y., Chaudhuri, S. (eds) NASA Formal Methods. NFM 2023. Lecture Notes in Computer Science, vol 13903. Springer, Cham. https://doi.org/10.1007/978-3-031-33170-1_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-33170-1_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-33169-5

  • Online ISBN: 978-3-031-33170-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation