Skip to main content
SpringerLink
Log in

Complexity of adaptive testing in scenarios defined extensionally

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

In this paper, we consider a testing setting where the set of possible definitions of the Implementation Under Test (IUT), as well as the behavior of each of these definitions in all possible interactions, are extensionally defined, i.e., on an element-by-element and case-by-case basis. Under this setting, the problem of finding the minimum testing strategy such that collected observations will necessarily let us decide whether the IUT is correct or not (i.e., whether it necessarily belongs to the set of possible correct definitions or not) is studied in four possible problem variants: with or without non-determinism; and with or without more than one possible definition in the sets of possible correct and incorrect definitions. The computational complexity of these variants is studied, and properties such as PSPACE-completeness and Log-APX-hardness are identified.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Lee D, Yannakakis M. Principles and methods of testing finite state machines-a survey. Proceedings of the IEEE, 1996, 84(8): 1090–1123

    Article  Google Scholar 

  2. Petrenko A. Fault model-driven test derivation from finite state models: annotated bibliography. In: Proceedings of the 4th Summer School on Modeling and Verification of Parallel Processes. 2001, 196–205

  3. Dorofeeva R, El-Fakih K, Maag S, Cavalli A R, Yevtushenko N. FSM-based conformance testing methods: a survey annotated with experimental evaluation. Information and Software Technology, 2010, 52(12): 1286–1297

    Article  Google Scholar 

  4. Tretmans J. Conformance testing with labelled transition systems: implementation relations and test generation. Computer Networks and ISDN Systems, 1996, 29(1): 49–79

    Article  Google Scholar 

  5. Tretmans J. Testing concurrent systems: a formal approach. In: Proceedings of the 10th International Conference on Concurrency Theory. 1999, 46–65

  6. Brinksma E, Tretmans J. Testing transition systems: an annotated bibliography. In: Proceedings of the 4th Summer School on Modeling and Verification of Parallel Processes. 2001, 187–195

  7. Springintveld J, Vaandrager F, D’Argenio P R. Testing timed automata. Theoretical Computer Science, 2001, 254(1–2): 225–257

    Article  MathSciNet  Google Scholar 

  8. Krichen M, Tripakis S. Black-box conformance testing for real-time systems. In: Proceedings of the 11th International SPIN Workshop on Model Checking of Software. 2004, 109–126

  9. Berrada I, Castanet R, Félix P, Salah A. Test case minimization for realtime systems using timed bound traces. In: Proceedings of the 18th IFIP TC 6/WG 6.1 International Conference on Testing of Communicating Systems. 2006, 289–305

  10. Merayo M, Núñez M, Rodríguez I. Extending EFSMs to specify and test timed systems with action durations and time-outs. IEEE Transactions on Computers, 2008, 57(6): 835–844

    Article  MathSciNet  Google Scholar 

  11. Stoelinga M, Vaandrager F. A testing scenario for probabilistic automata. In: Proceedings of the 30th International Colloquium on Automata, Languages and Programming. 2003, 464–477

  12. López N, Núñez M, Rodríguez I. Specification, testing and implementation relations for symbolic-probabilistic systems. Theoretical Computer Science, 2006, 353(1–3): 228–248

    Article  MathSciNet  Google Scholar 

  13. Efatmaneshnik M, Shoval S, Joiner K. System test architecture evaluation: a probabilistic modeling approach. IEEE Systems Journal, 2019, 13(4): 3651–3662

    Article  Google Scholar 

  14. Morell L J. A theory of fault-based testing. IEEE Transactions on Software Engineering, 1990, 16(8): 844–857

    Article  Google Scholar 

  15. Hierons R M. Comparing test sets and criteria in the presence of test hypotheses and fault domains. ACM Transactions on Software Engineering and Methodology, 2002, 11(4): 427–448

    Article  Google Scholar 

  16. Hierons R M. Verdict functions in testing with a fault domain or test hypotheses. ACM Transactions on Software Engineering and Methodology, 2009, 18(4): 14

    Article  Google Scholar 

  17. Rodríguez I, Merayo M G, Núñez M. HOTL: hypotheses and observations testing logic. The Journal of Logic and Algebraic Programming, 2008, 74(2): 57–93

    Article  MathSciNet  Google Scholar 

  18. Rodríguez I, Llana L, Rabanal P. A general testability theory: classes, properties, complexity, and testing reductions. IEEE Transactions on Software Engineering, 2014, 40(9): 862–894

    Article  Google Scholar 

  19. Rodríguez I, Rosa-Velardo F, Rubio F. Introducing complexity to formal testing. Journal of Logical and Algebraic Methods in Programming, 2020, 111: 100502

    Article  MathSciNet  Google Scholar 

  20. Kushik N, Yevtushenko N. Adaptive homing is in P. In: Proceedings of the 10th Workshop on Model-Based Testing. 2015, 73–78

  21. Yenigün H, Yevtushenko N, Kushik N. The complexity of checking the existence and derivation of adaptive synchronizing experiments for deterministic FSMs. Information Processing Letters, 2017, 127: 49–53

    Article  MathSciNet  Google Scholar 

  22. Kushik N, Yevtushenko N, Yenigun H. Reducing the complexity of checking the existence and derivation of adaptive synchronizing experiments for nondeterministic FSMs. In: Proceedings of the 4th International Workshop on DomAin Specific Model-Based AppRoaches to vErificaTion and validaTiOn. 2016, 83–90

  23. Petrenko A, Yevtushenko N. Adaptive testing of deterministic implementations specified by nondeterministic FSMs. In: Proceedings of the 23rd IFIP WG 6.1 International Conference on Testing Software and Systems. 2011, 162–178

  24. Petrenko A, Yevtushenko N. Adaptive testing of nondeterministic systems with FSM. In: Proceedings of the 15th IEEE International Symposium on High-Assurance Systems Engineering. 2014, 224–228

  25. van den Bos P, Vaandrager F. State identification for labeled transition systems with inputs and outputs. In: Proceedings of the 16th International Conference on Formal Aspects of Component Software. 2019, 191–212

  26. Bloem R, Fey G, Greif F, Könighofer R, Pill I, Riener H, Röck F. Synthesizing adaptive test strategies from temporal logic specifications. Formal Methods in System Design, 2019, 55(2): 103–135

    Article  Google Scholar 

  27. Rodríguez I. A general testability theory. In: Proceedings of the 20th International Conference on Concurrency Theory. 2009, 572–586

  28. Crescenzi P. A short guide to approximation preserving reductions. In: Proceedings of the 12th Annual IEEE Conference on Computational Complexity. 1997, 262–273

  29. Feige U. A threshold of ln n for approximating set cover. Journal of the ACM, 1998, 45(4): 634–652

    Article  MathSciNet  Google Scholar 

  30. Sipser M. Introduction to the Theory of Computation. Boston: Cengage Learning, 2012

    MATH  Google Scholar 

  31. Stockmeyer L. Classifying the computational complexity of problems. The Journal of Symbolic Logic, 1987, 52(1): 1–43

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dpto. Sistemas Informáticos y Computación, Facultad de Informática, Universidad Complutense de Madrid, Madrid, 28040, Spain

    Ismael Rodríguez & Fernando Rubio

  2. Instituto de Tecnologías del Conocimiento, Universidad Complutense de Madrid, Madrid, 28040, Spain

    Ismael Rodríguez & Fernando Rubio

  3. Instituto de Biomecánica de Valencia, Universitat Politècnica de València, València, 46022, Spain

    David Rubio

Authors
  1. Ismael Rodríguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Rubio
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fernando Rubio
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fernando Rubio.

Additional information

Ismael Rodríguez is an Assocciate Proffessor in the Computer Systems and Computation Department, Complutense University of Madrid, Spain. He obtained his MS degree in Computer Science in 2001 and his PhD in the same subject in 2004. He received the Best Thesis Award of his faculty in 2004. He has published more than 100 papers in international refereed conferences and journals. His research interests cover formal testing techniques, swarm and evolutionary optimization algorithms, computational complexity, formal methods, and functional programming.

David Rubio obtained a bachelor degree in Computer Science and another bachelor degree in Mathematics from Complutense University of Madrid, Spain in 2019. He has also studied a master on Artificial Intelligence at Universitat Politècnica de València, Spain, where he is currently a researcher at the Biomecanics Institute. His research interests cover image recognition, artificial vision, artificial intelligence, and formal testing techniques.

Fernando Rubio is an Associate Professor in the Computer Systems and Computation Department, Complutense University of Madrid, Spain. He obtained his MS degree in Computer Science in 1997, and he was awarded by the Spanish Ministry of Education with “Primer Premio Nacional Fin de Carrera”. He finished his PhD in the same subject four years later. He received the Best Thesis Award of his faculty in 2001. He has published more than 100 papers in international refereed conferences and journals. His research interests cover formal methods, swarm and evolutionary optimization methods, parallel computing, and functional programming.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rodríguez, I., Rubio, D. & Rubio, F. Complexity of adaptive testing in scenarios defined extensionally. Front. Comput. Sci. 17, 173206 (2023). https://doi.org/10.1007/s11704-022-1673-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11704-022-1673-9

Keywords

Search

Navigation