research-article

RTCM: a natural language based, automated, and practical test case generation framework

Authors:

Man ZhangAuthors Info & Claims

ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and Analysis

Pages 397 - 408

https://doi.org/10.1145/2771783.2771799

Published: 13 July 2015 Publication History

Abstract

Based on our experience of collaborating with industry, we observed that test case generation usually relies on test case specifications (TCSs), commonly written in natural language, specifying test cases of a System Under Test at a high level of abstraction. In practice, TCSs are commonly used by test engineers as reference documents to perform these activities: 1) Manually executing test cases in TCSs; 2) Manually coding test cases in a test scripting language for automated test case execution. In the latter case, the gap between TCSs and executable test cases has to be filled by test engineers, requiring a significant amount of coding effort and domain knowledge. Motivated by the above observations from the industry, we first propose, in this paper, a TCS language, named as Restricted Test Case Modeling (RTCM), which is based on natural language and composed of an easy-to-use template, a set of restriction rules and keywords. Second, we propose a test case generation tool (aToucan4Test), which takes TCSs in RTCM as input and generates either manual test cases or automatically executable test cases, based on various coverage criteria defined on RTCM. To assess the applicability of RTCM, we manually modeled two industrial case studies and examined 30 automatically generated TCSs. To evaluate aToucan4Test, we modeled three subsystems of a Video Conferencing System developed by Cisco Systems, Norway and automatically generated executable test cases. These test cases were successfully executed on two commercial software versions. In the paper, we also discuss our experience of applying RTCM and aToucan4Test in an industrial context and compare our approach with other model-based testing methodologies.

References

[1]

Algayres, B., Lejeune, Y., and Hugonnet, F., 2012. GOAL: Observing SDL behaviors with Geode. The Proceedings of SDL’95, 223-230.

[2]

Ali, S., Briand, L.C., and Hemmati, H., 2012. Modeling robustness behavior using aspect-oriented modeling to support robustness testing of industrial systems. Software & Systems Modeling 11, 4, 633-670.

Digital Library

[3]

Ali, S. and Hemmati, H., 2014. Model-based Testing of Video Conferencing Systems: Challenges, Lessons Learnt, and Results. In Software Testing, Verification and Validation (ICST), 2014 IEEE Seventh International Conference on IEEE, 353-362.

Digital Library

[4]

Ali, S., Hemmati, H., Holt, N.E., Arisholm, E., and Briand, L.C., 2010. Model transformations as a strategy to automate model-based testing-A tool and industrial case studies. Simula Research Laboratory, Technical Report (2010-01).

[5]

Ali, S., Zohaib Iqbal, M., Khalid, M., and Andrea, A., 2015. Improving the Performance of OCL Constraint Solving with Novel Heuristics for Logical Operations: A Search-Based Approach. Empirical Software Engineering.

[6]

Badri, M., Badri, L., and Naha, M., 2004. A use case driven testing process: Towards a formal approach based on UML collaboration diagrams. In Formal Approaches to Software Testing Springer, 223-235.

[7]

Barros, F.A., Neves, L., Hori, É., and Torres, D., 2011. The ucsCNL: A Controlled Natural Language for Use Case Specifications. In Seke, 250-253.

[8]

Beck, K., 2003. Test-driven development: by example. Addison-Wesley Professional.

Digital Library

[9]

Bittner, K., 2002. Use case modeling. Addison-Wesley Longman Publishing Co., Inc.

Digital Library

[10]

Bochmann, G.V., Petrenko, A., Bellal, O., and Maguiraga, S., 1997. Automating the process of test derivation from SDL specifications. Sdl 97, 22, 261-276.

[11]

Brömstrup, L. and Hogrefe, D., 1995. TESDL: experience with generating test cases from SDL specifications. In Conformance testing methodologies and architectures for OSI protocols IEEE Computer Society Press, 455-467.

Digital Library

[12]

Bruegge, B. and Dutoit, A.H., 2004. Object-Oriented Software Engineering Using UML, Patterns and Java- (Required). Prentice Hall.

Digital Library

[13]

Chelimsky, D., Astels, D., Helmkamp, B., North, D., Dennis, Z., and Hellesoy, A., 2010. The RSpec book: Behaviour driven development with Rspec, Cucumber, and friends. Pragmatic Bookshelf.

Digital Library

[14]

Cucumber, 2014. cucumber: Simple, human collaboration. https://cucumber.io/.

[15]

Dssouli, R., Saleh, K., Aboulhamid, E., En-Nouaary, A., and Bourhfir, C., 1999. Test development for communication protocols: towards automation. Computer Networks 31, 17, 1835-1872.

Digital Library

[16]

Fitnesse, 2015. FitNesse. http://fitnesse.org/FitNesse.

[17]

Guo, J., White, J., Wang, G., Li, J., and Wang, Y., 2011. A genetic algorithm for optimized feature selection with resource constraints in software product lines. Journal of Systems and Software 84, 12, 2208-2221.

Digital Library

[18]

Hametner, R., Winkler, D., and Zoitl, A., 2012. Agile testing concepts based on keyword-driven testing for industrial automation systems. In IECON 2012-38th Annual Conference on IEEE Industrial Electronics Society IEEE, 3727-3732.

[19]

Hesari, S., Behjati, R., and Yue, T., 2013. Towards a systematic requirement-based test generation framework: industrial challenges and needs. In Requirements Engineering Conference (RE), 2013 21st IEEE International IEEE, 261-266.

[20]

Iso/Iec/Ieee, 2013. Software and systems engineering-Software testing-Part 1: Concepts and definitions. ISO/IEC/IEEE 29119-1:2013(E), Switzerland.

[21]

Little, G. and Miller, R.C., 2006. Translating keyword commands into executable code. In Proceedings of the 19th annual ACM symposium on User interface software and technology ACM, 135-144.

Digital Library

[22]

Mugridge, R. and Cunningham, W., 2005. Fit for developing software: framework for integrated tests. Pearson Education.

Digital Library

[23]

Myers, G.J., Sandler, C., and Badgett, T., 2011. The art of software testing. John Wiley & Sons.

Digital Library

[24]

Nebut, C., Fleurey, F., Le Traon, Y., and Jezequel, J., 2003. Requirements by contracts allow automated system testing. In Software Reliability Engineering, 2003. ISSRE 2003.

Digital Library

[25]

14th International Symposium on IEEE, 85-96.

[26]

North, D., 2006.

[27]

Introducing behaviour driven development. Better Software.

[28]

North, D., 2007. Introducing RBehave. Nov., 2014. http://dannorth.net/2007/06/17/introducing-rbehave/.

[29]

North, D., 2007. What's in a story? Nov., 2014. http://dannorth.net/whats-in-a-story/.

[30]

Omg, 2014. Meta Object Facility (MOF) Core Specification (Version 2.4.2).

[31]

Ryser, J. and Glinz, M., 1999. A scenario-based approach to validating and testing software systems using statecharts. In Proc. 12th International Conference on Software and Systems Engineering and their Applications Citeseer.

[32]

Tahat, L.H., Vaysburg, B., Korel, B., and Bader, A.J., 2001. Requirement-based automated black-box test generation. In Computer Software and Applications Conference, 2001. COMPSAC 2001. 25th Annual International IEEE, 489-495.

Digital Library

[33]

Tang, J., Cao, X., and Ma, A., 2008. Towards adaptive framework of keyword driven automation testing. In Automation and Logistics, 2008. ICAL 2008. IEEE International Conference on IEEE, 1631-1636.

[34]

Thummalapenta, S., Sinha, S., Singhania, N., and Chandra, S., 2012. Automating test automation. In Software Engineering (ICSE), 2012 34th International Conference on IEEE, 881-891.

Digital Library

[35]

Wang, G., Pascucci, V., Goknil, A., Braind, L., and Lqbal, Z., 2015. Automatic Generation of System Test Cases from Use Case Specifications. In Proceedings of the International Symposium on Software Testing and Analysis (Baltimore, Maryland2015).

Digital Library

[36]

Yue, T., Briand, L., and Labiche, Y., 2014. aToucan: An Automated Framework to Derive UML Analysis Models from Use Case Models. ACM Transactions on Software Engineering and Methodology.

Digital Library

[37]

Yue, T., Briand, L.C., and Labiche, Y., 2013. Facilitating the transition from use case models to analysis models: Approach and experiments. ACM Transactions on Software Engineering and Methodology (TOSEM) 22, 1, 5.

Digital Library

[38]

Zhang, G., Yue, T., Wu, J., and Ali, S., 2013. Zen-RUCM: A Tool for Supporting a Comprehensive and Extensible Use Case Modeling Framework. In Demos/Posters /StudentResearch@ MoDELS Citeseer, 41-45.

[39]

Zhang, M., Yue, T., Ali, S., Zhang, H., and Wu, J., 2014. A Systematic Approach to Automatically Derive Test Cases from Use Cases Specified in Restricted Natural Languages. In System Analysis and Modeling: Models and Reusability Springer, 142-157.

Cited By

De Vito GVassallo GPalomba FFerrucci F(2024)AGORA: An Approach for Generating Acceptance Test Cases from Use Cases2024 50th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)10.1109/SEAA64295.2024.00027(126-133)Online publication date: 28-Aug-2024
https://doi.org/10.1109/SEAA64295.2024.00027
Wang GWu JYang HSun QYue T(2024)Test Architecture Generation by Leveraging BERT and Control and Data FlowsEngineering of Complex Computer Systems10.1007/978-3-031-66456-4_7(125-145)Online publication date: 29-Sep-2024
https://doi.org/10.1007/978-3-031-66456-4_7
Viggiato MPaas DBuzon CBezemer C(2023)Identifying Similar Test Cases That Are Specified in Natural LanguageIEEE Transactions on Software Engineering10.1109/TSE.2022.317027249:3(1027-1043)Online publication date: 1-Mar-2023
https://doi.org/10.1109/TSE.2022.3170272
Show More Cited By

Index Terms

RTCM: a natural language based, automated, and practical test case generation framework
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Achieving scalable mutation-based generation of whole test suites

Without complete formal specification, automatically generated software tests need to be manually checked in order to detect faults. This makes it desirable to produce the strongest possible test set while keeping the number of tests as small as ...
An orchestrated survey of methodologies for automated software test case generation

Test case generation is among the most labour-intensive tasks in software testing. It also has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has been one of the most active research topics in software ...
An automated technique for risk-based test case generation and prioritization
AST '08: Proceedings of the 3rd international workshop on Automation of software test

In practice, available testing budgets limit the number of test cases that can be executed. Thus, a representative subset of all possible test cases must be chosen to guarantee adequate coverage of a test object. In risk-based testing, the probability ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and Analysis

July 2015

447 pages

ISBN:9781450336208

DOI:10.1145/2771783

General Chair:
Michal Young
University of Oregon, USA
,
Program Chair:
Tao Xie
University of Illinois at Urbana-Champaign, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '15

Sponsor:

SIGSOFT

ISSTA '15: International Symposium on Software Testing and Analysis

July 13 - 17, 2015

MD, Baltimore, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
840
Total Downloads

Downloads (Last 12 months)61
Downloads (Last 6 weeks)5

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

De Vito GVassallo GPalomba FFerrucci F(2024)AGORA: An Approach for Generating Acceptance Test Cases from Use Cases2024 50th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)10.1109/SEAA64295.2024.00027(126-133)Online publication date: 28-Aug-2024
https://doi.org/10.1109/SEAA64295.2024.00027
Wang GWu JYang HSun QYue T(2024)Test Architecture Generation by Leveraging BERT and Control and Data FlowsEngineering of Complex Computer Systems10.1007/978-3-031-66456-4_7(125-145)Online publication date: 29-Sep-2024
https://doi.org/10.1007/978-3-031-66456-4_7
Viggiato MPaas DBuzon CBezemer C(2023)Identifying Similar Test Cases That Are Specified in Natural LanguageIEEE Transactions on Software Engineering10.1109/TSE.2022.317027249:3(1027-1043)Online publication date: 1-Mar-2023
https://doi.org/10.1109/TSE.2022.3170272
Kim MSinha SOrso ABissyandé TKlein JBird CSarro F(2023)Adaptive REST API Testing with Reinforcement LearningProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE56229.2023.00218(446-458)Online publication date: 11-Nov-2023
https://dl.acm.org/doi/10.1109/ASE56229.2023.00218
Coutinho JAndrade WMachado P(2022)A Survey of Requirements Engineering and Software Testing Practices in Agile TeamsProceedings of the 7th Brazilian Symposium on Systematic and Automated Software Testing10.1145/3559744.3559746(9-18)Online publication date: 3-Oct-2022
https://dl.acm.org/doi/10.1145/3559744.3559746
Wang CPastore FGoknil ABriand L(2022)Automatic Generation of Acceptance Test Cases From Use Case Specifications: An NLP-Based ApproachIEEE Transactions on Software Engineering10.1109/TSE.2020.299850348:2(585-616)Online publication date: 1-Feb-2022
https://doi.org/10.1109/TSE.2020.2998503
Liu KWu JYang HSun QWan R(2022)SRTEF: Test Function Recommendation With Scenarios and Latent Semantic for Implementing Stepwise Test CaseIEEE Transactions on Reliability10.1109/TR.2022.316464571:2(1127-1140)Online publication date: Jun-2022
https://doi.org/10.1109/TR.2022.3164645
Yu YWang TYue T(2021)Specifying Autonomous Driving Scenarios2021 IEEE 29th International Requirements Engineering Conference Workshops (REW)10.1109/REW53955.2021.00079(439-444)Online publication date: Sep-2021
https://doi.org/10.1109/REW53955.2021.00079
Liu KWu JYang HSun QWan R(2021)SRTEF: Automatic Test Function Recommendation with Scenarios for Implementing Stepwise Test Case2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS54544.2021.00115(1069-1078)Online publication date: Dec-2021
https://doi.org/10.1109/QRS54544.2021.00115
Shi YLu CZhang MZhang HYue TAli S(2021)Restricted Natural Language and Model-based Adaptive Test Generation for Autonomous Driving2021 ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems (MODELS)10.1109/MODELS50736.2021.00019(101-111)Online publication date: Oct-2021
https://doi.org/10.1109/MODELS50736.2021.00019
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten