Abstract
HOTTest is a model based test automation technique of software systems based on models of the system described using HaskellDB. HaskellDB is an embedded domain specific language derived from Haskell. HOTTest enforces a systematic abstraction process and exploits system invariants for automatically producing test cases for domain specific requirements. Use of functional languages for system modeling is a new concept and hence HOTTest is subject to concerns of usability, like any other new technique. Also, the syntax and the declarative style of Haskell based languages make them difficult to learn. Similar concerns can be raised for HOTTest as it shares the same syntax with Haskell. In this paper we describe an experiment designed to study the usability of HOTTest and to compare it with existing model based test design techniques. The results show that HOTTest is more usable than the traditional technique and demonstrate that the test suites produced by HOTTest are more effective and efficient than those generated using the traditional model based test design technique.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Arango G (1989) Domain analysis: from art form to engineering discipline. Proc. 5th International Workshop on Software Specification and Design, Pittsburgh, pp 152–159
Barnett M, Grieskamp W, Nachmanson L, Schulte W, Tillmann N, and Veanes M (2003). Modelbased testing with asmL.NET. In Proc. 1st European Conference on Model-Driven Software Engineering
Basili V (1995) Software quality assurance and measurement: a worldwide perspective, applying the goal/question/metric paradigm in the experience factory, Chapter 2, International Thomson Computer Press, ITP An International Thomson Publishing Company, pp 21–44
Bjesse P, Claessen K, Sheeran M, and Singh S (1998, September) Lava: hardware design in Haskell. In Proc. ICFP
Campbell DT, and Stanley JC (1963) Experimental and quasi-experimental designs for research. Rand McNally and Company, Chicago
Chaar JK, Halliday MJ, Bhandari IS, and Chillarege R (1993) In-process evaluation of software inspection and test. IEEE Trans Softw Eng 19(11):1055–1070
Chays D, Dan S, Frankl PG, Vokolos FI, and Weyuker EJ (2000) A framework for testing database applications. In Proc. ISSTA, Portland, pp 147–157
Cohen J (1990) A power primer. Psychol Bull 112(1):155–159
Deursen A, Klint P, and Visser J (2000) Domain-specific languages: an annotated bibliography. ACM SIGPLAN Notices 35(6):26–36, http://www.cwi.nl/projects/dsl.
Dick J, and Faivre A (1993) Automating the generation and sequencing of test cases from modelbased specifications. In Proc. First International Symposium of Formal Methods Europe on Industrial-Strength Formal Methods, Odense, pp 268–284
Dssouli R, Saleh K, Aboulhamid E, En-Nouaary A, and Bourhfir C (1999) Test development for communication protocols: towards automation. Comput Networks, 31:1835–1872
Field A, and Hole G (2003) How to design and report experiments. SAGE Publications, London, UK
Hall AV (1991) Relations between specifications and testing. Inf Softw Technol 33(1):47–52
HaskellDB Manual (2002) Galois Connections Inc.
Huber JT (1999) Efficiency and effectiveness measures to help guide the business of software testing. Applications of Software Measurement, HP Labs Research Report [Online]. Available: http://www.benchmarkqa.com/PDFs/efficiency_measures.pdf
Hudak P (2000) The Haskell school of expression. Cambridge University Press, New York
Hughes A, and Grawoig D (1971) Statistics: a foundation for analysis. Addition Wesley Publishing Company, Reading, MA
IEEE Guide to Software Requirements Specification (1984) IEEE Standard 830
Jagadeesan LJ, Porter A, Puchol C, Ramming CJ, and Votta LG (1997) Specification-based testing of reactive software: tools and experiments. In Proc. 19th International Conference on Software Engineering, Boston, pp 525–535
John BE (1996) Evaluating usability evaluation techniques. ACM Computing Surveys, [Online] 28 (4es). Available: http://www.acm.org/pubs/citations/journals/surveys/1996-28-4es/a139-john/
Leijen D, and Meijer E (1999) Domain specific embedded compilers. Proc. 2nd USENIX Conference on DSL, Austin, pp 109–122
Memon AM, Pollack ME, and Soffa ML (2001) Hierarchical GUI test case generation using automated planning. IEEE Trans Softw Eng 27(2):144–155
Nielsen J (1993) Usability engineering. Academic Press Inc, San Diego, CA
Peter DK, and Parnas DL (1998) Using test oracles generated from program documentation. IEEE Trans Softw Eng 24(3):161–173
Rational XDE Tester User_s Guide (2004) IBM Corporation, New York, NY
Savage P, Walters S, and Stephenson M (1997) Automated test methodology for operational flight programs. In Proc. IEEE Aerospace Conference, vol. 4, 293–305
Silktest User_s Guide, Version 6.5 (2002) Segue Software Inc., Lexington, MA
Sinha A, Smidts C, and Moran A (2003) Enhanced testing of domain specific applications by automatic extraction of axioms from functional specifications. In Proc. 14th International Symposium on Software Reliability Engineering, Denver, pp 181–190
Sinha A (2005) Domain specific test case generation using higher ordered typed languages for specification. PhD Thesis, University of Maryland
Stark GE, Durst RC, and Pelnik TM (2001) An evaluation of software testing metrics for NASA_s mission control center” MITRE
Stocks PA, and Carrington DA (1991) Deriving software test cases from formal specifications. In Proc. 6th Australian Software Engineering Conference, Sydney, pp 327–340
Test Master User_s Guide (1999) Release 1.9.5, Empirix Inc., New Hampshire
Testing Tool Information, Grove Consultants, 2002
Thiemann P, An Embedded Domain-Specific Language for Type-Safe Server-Side Web-Scripting. [Online] http://www.informatik.uni-freiburg.de/~thiemann/haskell/WASH/draft.pdf
Thompson S (1999) Haskell: the craft of functional programming. Addison-Wesley, Essex
Tretmans J, and Belinfante A (1999) Automatic testing with formal methods. In Proc. Euro-STAR'99: 7th European Int. Conference on Software Testing, Analysis & Review, Barcelona, Spain, EuroStar Conferences, Galway, Ireland, pp 8–12
Vaysburg B, Tahat LH, and Korel B (2002) Dependence analysis in reduction of requirement based test suites. In Proc. International Symposium on Software Testing and Analysis, Roma, Italy, pp 107–111
Wang CJ, and Liu MT (1993) Generating test cases for EFSM with given fault models. In Proc. IEEE Infocom, vol. 2, 774–781
Williams CE (2001) Toward a test-ready meta-model for use cases. In Proc. Workshop on Practical UML-based Rigorous Development Methods, Toronto, CA, pp 270–287
WinRunner User's Guide, Version 7.01 (2001) Mercury Interactive Inc., Sunnyvale, CA
Author information
Authors and Affiliations
Corresponding author
Additional information
Editor: James Miller
Rights and permissions
About this article
Cite this article
Sinha, A., Smidts, C. An experimental evaluation of a higher-ordered-typed-functional specification-based test-generation technique. Empir Software Eng 11, 173–202 (2006). https://doi.org/10.1007/s10664-006-6401-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-006-6401-9