Skip to main content
SpringerLink
Log in

A requirements inspection method based on scenarios generated by model mutation and the experimental validation

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

The requirements phase is the most critical phase of the software development life cycle. The quality of the requirements specification affects the overall quality of the subsequent phases and hence, the software product. An effective and efficient method to qualify the software requirements specification (SRS) is necessary to ensure the reliability and safety of software. In this paper, a requirements inspection method based on scenarios generated by model mutation (RIMSM) is proposed to detect defects in the functional requirements of a safety-critical system. The RIMSM method models software requirements using a High Level Extended Finite State Machine (HLEFSM). A method that executes the HLEFSM model is defined. The method uncovers the behaviors and generates the outputs of the system for a given scenario. To identify an adequate set of scenarios in which the model shall be executed, an analogue to mutation testing is defined which applies to the requirements phase. Twenty-one mutation operators are designed based on a taxonomy of defects defined for the requirements phase. Mutants of the HLEFSM model are generated using these operators. Further, an algorithm is developed to identify scenarios that can kill the mutants. The set of scenarios is considered to be adequate for detecting defects in the model when all mutants generated are killed. The HLEFSM model is then executed for the scenarios generated. The results of execution are used to detect defects in the model. A Requirements Inspection Tool based on Scenarios Generated by Model Mutation (RITSM) is developed to automate the application of the RIMSM method. The performance and usability of the RIMSM method are studied and demonstrated in an experiment by comparing the RIMSM method to the checklist-based reading method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29

Similar content being viewed by others

References

  • Agrawal H, DeMillo RA, Hathaway B, Hsu W, Wynne H, Krauser EW, Martin RJ, Mathur AP, Spafford E. (1989) “Design of Mutant Operators for the C programming language.” Technical report SERC-TR-41-P, Software Engineering Research Center, Purdue University

  • Ali SW, Ahmed QA, Shafi I. (2018). “Process to Enhance the Quality of Software Requirement Specification Document.” 2018 International Conference on Engineering and Emerging Technologies

  • Alshazly AA, Elfatatry AM, Abougabal MS (2014) Detecting defects in software requirements specification. Alexandria Engineering Journal 53(3):513–527

    Article  Google Scholar 

  • Arndt SA, Alvarado R, Dittman B, Mott K, Wood R (2017) “NRC Technical Basis For Evaluation Of Its Position On Protection Against Common Cause Failure In Digital Systems Used In Nuclear Power Plants.” In Proceedings of 2017 NPIC-HMIT

  • Budd TA, Gopal AS (1985) Program testing by specification mutation. Comput Lang 10(1):63–73. https://doi.org/10.1016/0096-0551(85)90011-6

    Article  MATH  Google Scholar 

  • Budd TA, DeMillo RA, Lipton RJ, Sayward FG (1978) The Design of a Prototype Mutation System for program testing. Proceedings of the AFIPS National Computer Conference 74:623–627

    Google Scholar 

  • Cisneros A, Rafael J, Garcia GDLR, and Fernandez-Y-Fernandez CA (2018) “Software Requirement Specification for the Automotive Sector: The Case of a Post-Collision Event Control System.” Proceedings of 5th International Conference in Software Engineering Research and Innovation, IEEE

  • Dalpiaz F, van der Schalk I, Brinkkemper S, Aydemir FB, Lucassen G (2019) Detecting terminological ambiguity in user stories: tool and experimentation. Inf Softw Technol 110:3–16. https://doi.org/10.1016/j.infsof.2018.12.007

    Article  Google Scholar 

  • Demillo RA, Lipton RJ, Sayward FG (1978) Hints on test data selection : help for the practicing programmer. Computer 11(4):33–41

    Article  Google Scholar 

  • Derenthal FE, Elks CR, Bakker T, and Fotouhi M (2017) “Virtualized Hardware Environments for Supporting Digital I&C Verification.” Proceedings of 2017 NPIC-HMIT, 1658–70

  • Fabbri SCPF, Maldonado JC, Sugeta T, Masiero PC (1999) Mutation Testing Applied to Validate Specifications Based on Statecharts. Proceedings of 10th International Symposium on Software Reliability Engineering, IEEE:210–219. https://doi.org/10.1109/ISSRE.1999.809326

  • Hamlet RG (1977) Testing programs with the aid of a compiler. IEEE Trans Softw Eng 4:279–290

    Article  Google Scholar 

  • He L, Carver J (2006) PBR vs. Checklist: A Replication in the N-Fold Inspection Context. Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engineering:95–104. https://doi.org/10.1145/1159733.1159750

  • Hierons RM, Merayo MG (2007) “Mutation testing from probabilistic finite state machines.” Proceedings of Testing: Academic and Industrial Conference Practice and Research Techniques, 141–50. https://doi.org/10.1109/TAIC.PART.2007.20

  • Hierons R, Merayo M (2009) Mutation testing from probabilistic and stochastic finite state machines. J Syst Softw 82(11):1804–1818. https://doi.org/10.1016/j.jss.2009.06.030

    Article  Google Scholar 

  • IEEE (2018) Systems and software engineering-life cycle processes: requirements engineering. ISO/IEC/IEEE 29148:2018

    Google Scholar 

  • Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng 37(5):649–678

    Article  Google Scholar 

  • Kim SW, Clark J, McDermid J (1999) “The rigorous generation of Java mutation operators using HAZOP.” Proceedings of the 12th international conference on software and systems engineering and their applications ICSSEA99, 1–20

  • King KN, Jefferson Offutt A (1991) A Fortran language system for mutation-based software testing. Software: Practice and Experience 21(7):685–718. https://doi.org/10.1002/spe.4380210704

    Article  Google Scholar 

  • Kirby J (1987) “Software requirements for an automobile cruise control and monitoring system.” NRL-SCR

  • Laitenberger O (2002) A survey of software inspection technologies. Handbook Softw Eng Knowl Eng 2:517–555

    Article  Google Scholar 

  • Lanubile F, Visaggio G (2000) “Evaluating Defect Detection Techniques for Software Requirements Inspections.” International Software Engineering Research Network Report No. 00–08

  • LeBlanc D, Bezzina D, Tiernan T, Gabel M, Pomerleau D (2008) “Functional requirements for integrated vehicle-based safety system (IVBSS).” The University of Michigan Transportation Research Institute, Visteon Corporation and Cognex Corporation

  • Lee S-M, Chao A (1994) Estimating population size via sample coverage for closed capture-recapture models. Biometrics 50(1):88–97

    Article  Google Scholar 

  • Li B, Smidts C, Huang F, Quinn T (2018) “A Quantification Framework for Software Safety in the Requirements Phase: Application to Nuclear Power Plants.” In Proceedings of 2018 Annual Reliability and Maintainability Symposium (RAMS)

  • Li B, Smidts CS (2017) “Extension of Mutation Testing for the Requirements and Design Faults.” In Proceedings of 2017 NPIC-HMIT

  • Li X, and Gupta J (2013) “ARPS: An Automated Reliability Prediction System Tool for Safety Critical Software.” PSA, 22–27

  • Li X, Mutha C, Smidts CS (2016) An automated software reliability prediction system for safety critical software. Empir Softw Eng 21(6):2413–2455. https://doi.org/10.1007/s10664-015-9412-6

    Article  Google Scholar 

  • Lipton RJ (1971) “Fault diagnosis of computer programs.” Student Report, Carnegie Mellon University

  • Lorber F, Larsen KG, Nielsen B (2018) “Model-based mutation testing of real-time systems via model checking.” Proceedings of 2018 IEEE 11th international conference on software testing, verification and validation workshops (ICSTW), 59–68. https://doi.org/10.1109/ICSTW.2018.00029

  • Ma YS, Offutt J, Kwon YR (2005) MuJava: an automated class mutation system. Software Testing Verification and Reliability 15(2):97–133. https://doi.org/10.1002/stvr.308

    Article  Google Scholar 

  • McHugh ML (2012b) Interrater reliability: the kappa statistic. Biochemia Medica: Biochemia Medica 22(3)

  • De Moura L, Bjørner N, Bjorner N (2008) Z3: an efficient SMT solver. In: International conference on tools and algorithms for the construction and analysis of systems, pp 337–340. https://doi.org/10.1007/978-3-540-78800-3_24

    Chapter  Google Scholar 

  • NCSS Statistical Software (2020) “Mann-Whitney U or Wilcoxon Rank-Sum Tests.” NCSS, LLC. Kaysville, Utah, USA, 503–15

  • Okun V. 2004. “Specification mutation for test generation and analysis.” Dissertation, University of Maryland Baltimore County

  • Pinto Ferraz Fabbri SC, Delamaro ME, Maldonado JC, Masiero PC (1994) Mutation Analysis Testing for Finite State Machines. Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering:220–229. https://doi.org/10.1109/ISSRE.1994.341378

  • Porter AA, Votta LG, Basili VR (2002) Comparing detection methods for software requirements inspections: a replicated experiment. IEEE Trans Softw Eng 21(6):563–575. https://doi.org/10.1109/32.391380

    Article  Google Scholar 

  • Santoso S, Sudarno R, Maerani JS, Cahyono A (2019) Software requirement analysis for digital based reactor protection system of RDE design. J Phys Conf Ser 1198(2). https://doi.org/10.1088/1742-6596/1198/2/022046

  • Sawilowsky SS (2009) New Effect Size Rules of Thumb. Journal of Modern Applied Statistical Methods 8(2):597–599. https://doi.org/10.22237/jmasm/1257035100

    Article  MathSciNet  Google Scholar 

  • Staron M, Kuzniarz L, Thurn C (2005) An empirical assessment of using stereotypes to improve Reading techniques in software inspections. Proceedings: international conference on software engineering:63–69. https://doi.org/10.1145/1083292.1083308

  • Strobl F, Wisspeintner A. 1999. “Specication of an elevator control system.” Technische Universitat Munchen

  • Sugeta T, Maldonado JC, Eric Wong W (2004) Mutation testing applied to validate SDL specifications. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 2978:193–208. https://doi.org/10.1007/978-3-540-24704-3_13

    Article  Google Scholar 

  • Sullivan GM, Feinn R (2012) Using effect size—or why the P value is not enough. Journal of Graduate Medical Education 4(3):279–282. https://doi.org/10.4300/jgme-d-12-00156.1

    Article  Google Scholar 

  • Yang XM. 2007. “Towards a self-evolving software defect detection process.” Dissertation, University of Saskatchewan

  • Zhang H, Yue T, Ali S, Liu C (2016) Towards mutation analysis for use cases. Proceedings - 19th ACM/IEEE international conference on model driven engineering languages and systems, MODELS 2016:363–373. https://doi.org/10.1145/2976767.2976784

  • Zhang Z, Li K, Yuan L, Guanhua Y (2019) Mutation Model-Based Test Case Generation of Chinese Train Control System with Automatic Train Operation Function. Proceedings of 2018 International Conference on Intelligent Rail Transportation (ICIRT). https://doi.org/10.1109/ICIRT.2018.8641582

Download references

Acknowledgments

This material is based upon work supported by the U.S. Department of Energy, Office of Nuclear Energy, Nuclear Energy Enabling Technologies (NEET) program under the Award Number DE-NE0008434. This research is performed using funding received from the DOE Office of Nuclear Energy’s Nuclear Energy University Program under the Award Number DE-NE308896. This research is also funded by the Department of Mechanical and Aerospace Engineering, Ohio State University.

Author information

Authors and Affiliations

  1. Department of Mechanical and Aerospace Engineering at the Ohio State University, Columbus, OH, USA

    Boyuan Li, Xiaoxu Diao, Wei Gao & Carol Smidts

Authors
  1. Boyuan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoxu Diao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Carol Smidts
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Boyuan Li.

Additional information

Communicated by: Jeff Offutt

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, B., Diao, X., Gao, W. et al. A requirements inspection method based on scenarios generated by model mutation and the experimental validation. Empir Software Eng 26, 108 (2021). https://doi.org/10.1007/s10664-021-10001-9

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-021-10001-9

Keywords

Search

Navigation