Abstract
While traditional testing can give evidence that an implementation conforms to a specification there is no guarantee that the implementation therefore also satisfies user requirements. Consequently, it is useful to derive test cases also from requirement properties. In this paper we introduce two new test criteria to measure adequacy of existing test suites and to derive test cases with regard to requirements formalized in linear temporal logic (LTL). The first criterion is complementary to existing criteria for requirement properties in that it focuses on cases where literals in a property should not affect the outcome of the property. The second criterion uses mutation to measure the sensitivity of a test suite with regard to the implemented requirements. In addition to coverage measurement these criteria can also be used to automatically create coverage adequate test suites with a model checker. The results of a case study illustrate the feasibility of the new criteria and demonstrate that they complement existing criteria.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Pnueli, A.: The Temporal Logic of Programs. In: 18th Annual Symposium on Foundations of Computer Science, October 31-November 2, pp. 46–57. IEEE, Providence (1977)
Tan, L., Sokolsky, O., Lee, I.: Specification-Based Testing with Linear Temporal Logic. In: Proceedings of IEEE International Conference on Information Reuse and Integration (IRI 2004), pp. 493–498 (2004)
Whalen, M.W., Rajan, A., Heimdahl, M.P., Miller, S.P.: Coverage Metrics for Requirements-Based Testing. In: ISSTA 2006: Proceedings of the 2006 International Symposium on Software Testing and Analysis, pp. 25–36. ACM Press, New York (2006)
Chilenski, J.J., Miller, S.P.: Applicability of modified condition/decision coverage to software testing. Software Engineering Journal, 193–200 (September 1994)
Ammann, P., Offutt, J.: Introduction to Software Testing. Cambridge University Press, New York (2008)
Vilkomir, S.A., Bowen, J.P.: Reinforced Condition/Decision Coverage (RC/DC): A New Criterion for Software Testing. In: Bert, D., Bowen, J.P., Henson, M.C., Robinson, K. (eds.) B 2002 and ZB 2002. LNCS, vol. 2272, pp. 291–308. Springer, Heidelberg (2002)
Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking, 1st edn. MIT Press, Cambridge (2001) (3rd printing)
Fraser, G., Wotawa, F., Ammann, P.E.: Testing with model checkers: a survey. In: Software Testing, Verification and Reliability (to appear, 2009)
Beer, I., Ben-David, S., Eisner, C., Rodeh, Y.: Efficient Detection of Vacuity in ACTL Formulas. In: Grumberg, O. (ed.) CAV 1997. LNCS, vol. 1254, pp. 279–290. Springer, Heidelberg (1997)
Kupferman, O., Vardi, M.Y.: Vacuity Detection in Temporal Model Checking. In: Pierre, L., Kropf, T. (eds.) CHARME 1999. LNCS, vol. 1703, pp. 82–96. Springer, Heidelberg (1999)
Purandare, M., Somenzi, F.: Vacuum Cleaning CTL Formulae. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 485–499. Springer, Heidelberg (2002)
Ammann, P.E., Black, P.E., Majurski, W.: Using Model Checking to Generate Tests from Specifications. In: Proceedings of the Second IEEE International Conference on Formal Engineering Methods (ICFEM 1998), pp. 46–54. IEEE Computer Society, Los Alamitos (1998)
Gargantini, A.: Using Model Checking to Generate Fault Detecting Tests. In: Gurevich, Y., Meyer, B. (eds.) TAP 2007. LNCS, vol. 4454, pp. 189–206. Springer, Heidelberg (2007)
Acree, A.T., Budd, T.A., DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Mutation Analysis. Technical report, School of Information and Computer Science, Georgia Inst. of Technology, Atlanta, Ga (September 1979)
DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 34–41 (1978)
Ammann, P., Black, P.E.: A Specification-Based Coverage Metric to Evaluate Test Sets. In: HASE 1999: The 4th IEEE International Symposium on High-Assurance Systems Engineering, Washington, DC, USA, pp. 239–248. IEEE Computer Society, Los Alamitos (1999)
Havelund, K., Rosu, G.: Monitoring Programs Using Rewriting. In: ASE 2001: Proceedings of the 16th IEEE Int. Conference on Automated Software Engineering, Washington, DC, USA, p. 135. IEEE Computer Society, Los Alamitos (2001)
Black, P.E., Okun, V., Yesha, Y.: Mutation Operators for Specifications. In: Proceedings of the Fifteenth IEEE International Conference on Automated Software Engineering (ASE 2000), Washington, DC, USA. IEEE Computer Society Press, Los Alamitos (2000)
Cimatti, A., Clarke, E.M., Giunchiglia, F., Roveri, M.: NUSMV: A New Symbolic Model Verifier. In: Halbwachs, N., Peled, D.A. (eds.) CAV 1999. LNCS, vol. 1633, pp. 495–499. Springer, Heidelberg (1999)
Rajan, A., Whalen, M., Staats, M., Heimdahl, M.P.E.: Requirements Coverage as an Adequacy Measure for Conformance Testing. In: Liu, S., Maibaum, T., Araki, K. (eds.) ICFEM 2008. LNCS, vol. 5256, pp. 86–104. Springer, Heidelberg (2008)
Ammann, P., Ding, W., Xu, D.: Using a Model Checker to Test Safety Properties. In: Proceedings of the 7th International Conference on Engineering of Complex Computer Systems (ICECCS 2001), pp. 212–221. IEEE, Los Alamitos (2001)
Fraser, G., Wotawa, F.: Using Model-Checkers to Generate and Analyze Property Relevant Test-Cases. Software Quality Journal 16(2), 161–183 (2008)
Fraser, G., Ammann, P.: Reachability and propagation for ltl requirements testing. In: Proceedings of the 8th International Conference on Quality Software (QSIC 2008), pp. 189–198 (2008)
da Silva, D.A., Machado, P.D.L.: Towards Test Purpose Generation from CTL Properties for Reactive Systems. Electrical Notes in Theoretical Computer Science, vol. 164, pp. 29–40 (2006)
Machado, P.D.L., Silva, D.A., Mota, A.C.: Towards Property Oriented Testing. Electrical Notes in Theoretical Computer Science, vol. 184, pp. 3–19 (2007)
Rusu, V., Marchand, H., Tschaen, V., Jéron, T., Jeannet, B.: From safety verification to safety testing. Testing of Communicating Systems, 160–176 (2004)
Fernandez, J.C., Mounier, L., Pachon, C.: Property oriented test case generation. Formal Approaches to Software Testing, 147–163 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Fraser, G., Wotawa, F. (2009). Complementary Criteria for Testing Temporal Logic Properties. In: Dubois, C. (eds) Tests and Proofs. TAP 2009. Lecture Notes in Computer Science, vol 5668. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02949-3_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-02949-3_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-02948-6
Online ISBN: 978-3-642-02949-3
eBook Packages: Computer ScienceComputer Science (R0)