Skip to main content
Springer Nature Link
Log in

Testing software’s changing features with environment-driven abstraction identification

  • Original Article
  • Published:
Requirements Engineering Aims and scope Submit manuscript

Abstract

Abstractions are significant domain terms that have assisted in requirements elicitation and modeling. To extend the assistance toward requirements validation, we present in this paper an automated approach to identifying the abstractions for supporting requirements-based testing. We select relevant Wikipedia pages to serve as a domain corpus that is independent from any specific software system. We further define five novel patterns based on part-of-speech tagging and dependency parsing, and frame our candidate abstractions in the form of <key, value> pairs for better testability, where the “key” helps locate “what to test”, and the “value” helps guide “how to test it” by feeding in concrete data. We evaluate our approach with six software systems in two application domains: Electronic health records and Web conferencing. The results show that our abstractions are more accurate than those generated by a state-of-the-art technique. While the initial findings indicate our abstractions’ capabilities of revealing bugs and matching the environmental assumptions created manually, we articulate a new way to perform requirements-based testing by focusing on a software system’s changing features. Specifically, we hypothesize that the same feature would behave differently under a pair of opposing environmental conditions and assess our abstractions’ applicability to this new form of feature testing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available in the Zenodo repository, https://doi.org/10.5281/zenodo.5858384.

Notes

  1. http://www.pamelazave.com/faq.html (last accessed on 2022/09/14 12:07:09).

  2. Mathematical definition is provided later in Eq. (6) of Sect. 5.2.

  3. http://www.oxforddictionaries.com/definition/english/assumption (last accessed on 2022/09/14 12:07:09).

  4. iTrust is a Java application that provides patients with a means to keep up with their medical records and to communicate with their doctors [44].

  5. https://en.wikiquote.org/wiki/Edsger_W._Dijkstra (last accessed on 2022/09/14 12:07:09).

  6. By bug, we mean Eq. (4) is evaluated to be true (i.e., the entailment relationship fails to hold) under a specific set of \({\mathcal{E}}\), \({\rm{SUT}}\), and \({\mathcal{R}}\). We refine the notion of bug in Sect. 6 by formulating the test oracle without directly referring to \({\mathcal{R}}\) in order to address the distance between observing the software executions and validating the stakeholder goals.

  7. https://en.wikipedia.org/wiki/Electronic_health_record (last accessed on 2022/09/14 12:07:09).

References

  1. Gacitua R, Sawyer P, Gervasi V (2011) Relevance-based abstraction identification: technique and evaluation. Requir Eng 16(3):251–265

    Article  Google Scholar 

  2. Sawyer P, Rayson P, Cosh K (2005) Shallow knowledge as an aid to deep understanding in early phase requirements engineering. IEEE Trans Softw Eng 31(11):969–981

    Article  Google Scholar 

  3. Gacitua R, Sawyer P, Gervasi V (2010) On the effectiveness of abstraction identification in requirements engineering. In: Proceedings of the international requirements engineering conference (RE), Sydney, Australia, September–October 2010, pp 5–14

  4. Goldin L, Berry DM (1997) AbstFinder, a prototype natural language text abstraction finder for use in requirements elicitation. Autom Softw Eng 4(4):375–412

    Article  Google Scholar 

  5. Dwarakanath A, Ramnani RR, Sengupta S (2013) Automatic extraction of glossary terms from natural language requirements. In: Proceedings of the international requirements engineering conference (RE), Rio de Janeiro, Brazil, July 2013, pp 314–319

  6. Arora C, Sabetzadeh M, Briand LC, Zimmer F (2017) Automated extraction and clustering of requirements glossary terms. IEEE Trans Softw Eng 43(10):918–945

    Article  Google Scholar 

  7. Gemkow T, Conzelmann M, Hartig K, Vogelsang A (2018) Automatic glossary term extraction from large-scale requirements specifications. In: Proceedings of the international requirements engineering conference (RE), Banff, Canada, August 2018, pp 412–417

  8. Yu E (1997) Towards modeling and reasoning support for early-phase requirements engineering. In: Proceedings of the international symposium on requirements engineering (RE), Annapolis, MD, USA, January 1997, pp 226–235

  9. Ryan K (1993) The role of natural language in requirements engineering. In: Proceedings of the international symposium on requirements engineering (RE), San Diego, CA, USA, January 1993, pp 240–242

  10. Jin Z (2018) Environment modeling-based requirements engineering for software intensive systems. Morgan Kaufmann

    Google Scholar 

  11. Jackson M (1997) The meaning of requirements. Ann Softw Eng 3:5–21

    Article  Google Scholar 

  12. van Lamsweerde A (2009) Requirements engineering: from system goals to UML models to software specifications. Wiley

    Google Scholar 

  13. Tun TT, Lutz RR, Nakayama B, Yu Y, Mathur D, Nuseibeh B (2015) The role of environmental assumptions in failures of DNA nanosystems. In: Proceedings of the international workshop on complex faults and failures in large software systems (COUFLESS), Florence, Italy, May 2015, pp 27–33

  14. Knight JC (2002) Safety critical systems: challenges and directions. In: Proceedings of international conference on software engineering (ICSE), Orlando, Florida, USA, May 2002, pp 547–550

  15. Bhowmik T, Chekuri SR, Do AQ, Wang W, Niu N (2019) The role of environment assertions in requirements-based testing. In: Proceedings of the international requirements engineering conference (RE), Jeju Island, South Korea, September 2019, pp 75–85

  16. Peng Z, Rathod P, Niu N, Bhowmik T, Liu H, Shi L, Jin Z (2021) Environment-driven abstraction identification for requirements-based testing. In: Proceedings of the international requirements engineering conference (RE), Notre Dame, IN, USA, September 2021, pp 245–256

  17. Barr ET, Harman M, McMinn P, Shahbaz M, Yoo S (2015) The oracle problem in software testing: a survey. IEEE Trans Softw Eng 41(5):507–525

    Article  Google Scholar 

  18. Niu N, Mahmoud A (2012) Enhancing candidate link generation for requirements tracing: the cluster hypothesis revisited. In: Proceedings of the international requirements engineering conference (RE), Chicago, IL, USA, September 2012, pp 81–90

  19. Wang W, Gupta A, Niu N, Xu LD, Cheng J-RC, Niu Z (2018) Automatically tracing dependability requirements via term-based relevance feedback. IEEE Trans Ind Inf 14(1):342–349

    Article  Google Scholar 

  20. Wang W, Niu N, Liu H, Niu Z (2018) Enhancing automated requirements traceability by resolving polysemy. In: Proceedings of the international requirements engineering conference (RE), Banff, Canada, August 2018, pp 40–51

  21. Wermter J, Hahn U (2005) Finding new terminology in very large corpora. In: Proceedings of the international conference on knowledge capture (K-CAP), Banff, Canada, October 2005, pp 137–144

  22. Leveson NG (1995) Safeware: system safety and computers. Addison-Wesley

    Google Scholar 

  23. Hull E, Jackson K, Dick J (2010) Requirements engineering. Springer

    MATH  Google Scholar 

  24. Rahimi M, Xiong W, Cleland-Huang J, Lutz RR (2017) Diagnosing assumption problems in safety-critical products. In: Proceedings of the international conference on automated software engineering (ASE), Urbana, IL, USA, October–November 2017, pp 473–484

  25. Alenazi M, Niu N, Savolainen J (2020) A novel approach to tracing safety requirements and state-based design models. In: Proceedings of international conference on software engineering (ICSE), Seoul, South Korea, June–July 2020, pp 848–860

  26. Alrajeh D, Cailliau A, van Lamsweerde A (2020) Adapting requirements models to varying environments. In: Proceedings of international conference on software engineering (ICSE), Seoul, South Korea, June–July 2020, pp 50–61

  27. Yang C, Liang P, Avgeriou P (2018) Assumptions and their management in software development: a systematic mapping study. Inform Softw Technol 94:82–110

    Article  Google Scholar 

  28. Garlan D, Allen R, Ockerbloom J (2009) Architectural mismatch: why reuse is still so hard. IEEE Softw 26(4):66–69

    Article  Google Scholar 

  29. Jin X, Khatwani C, Niu N, Wagner M, Savolainen J (2016) Pragmatic software reuse in bioinformatics: how can social network information help? In: Proceedings of international conference on software reuse (ICSR), Limassol, Cyprus, June 2016, pp 247–264

  30. Bhuta J, Boehm B (2007) A framework for identification and resolution of interoperability mismatches in COTS-based systems. In: Proceedings of the international workshop on incorporating cots software into software systems: tools and techniques (IWICSS), Minneapolis, MN, USA, May 2007

  31. Bazaz A, Arthur JD, Tront JG (2006) Modeling security vulnerabilities: a constraints and assumptions perspective. In: Proceedings of the international symposium on dependable, autonomic and secure computing (DASC), Indianapolis, IN, USA, September–October 2006, pp 95–102

  32. Wang W, Dumont F, Niu N, Horton G (2020) Detecting software security vulnerabilities via requirements dependency analysis. IEEE Trans Softw Eng 48(5):1665–1675

    Article  Google Scholar 

  33. Kroll P, Kruchten P (2003) The rational unified process made easy: a practitioner’s guide to the RUP. Addison-Wesley

    Google Scholar 

  34. Laplante PA (2007) What every engineer should know about software engineering. CRC Press

    Book  Google Scholar 

  35. Unterkalmsteiner M, Gorschek T, Feldt R, Klotins E (2015) Assessing requirements engineering and software test alignment—five case studies. J Syst Softw 109:62–77

    Article  Google Scholar 

  36. Uusitalo EJ, Komssi M, Kauppinen M, Davis AM (2008) Linking requirements and testing in practice. In: Proceedings of the international requirements engineering conference (RE), Barcelona, Spain, September 2008, pp 265–270

  37. Flammini F, Mazzocca N, Orazzo A (2009) Automatic instantiation of abstract tests on specific configurations for large critical control systems. Softw Test Verif Reliab 19(2):91–110

    Article  Google Scholar 

  38. Miller T, Strooper PA (2012) A case study in model-based testing of specifications and implementations. Softw Test Verif Reliab 21(1):33–63

    Article  Google Scholar 

  39. de Santiago Júnior VA, Vijaykumar NL (2012) Generating model-based test cases from natural language requirements for space application software. Softw Test Verif Reliab 20(1):77–143

    Google Scholar 

  40. Siegl S, Hielscher K-S, German R (2010) Model based requirements analysis and testing of automotive systems with timed usage models. In: Proceedings of the international requirements engineering conference (RE), Sydney, Australia, September–October 2010, pp 345–350

  41. Garousi V, Bauer S, Felderer M (2020) NLP-assisted software testing: a systematic mapping of the literature. Inform Softw Technol 126:106 321:1–106 321:20

  42. Fischbach J, Vogelsang A, Spies D, Wehrle A, Junker M, Freudenstein D (2020) SPECMATE: automated creation of test cases from acceptance criteria. In: Proceedings of the international conference on software testing, validation and verification (ICST), Porto, Portugal, October 2020, pp 321–331

  43. Skoković P, Rakić-Skoković M (2010) Requirements-based testing process in practice. Int J Ind Eng Manag 1(4):155–161

    Google Scholar 

  44. Meneely A, Smith B, Williams L (2012) iTrust electronic health care system: a case study. In: Cleland-Huang J, Gotel O, Zisman A (eds) Software and systems traceability. Springer

    Google Scholar 

  45. Niu N, Brinkkemper S, Franch X, Partanen J, Savolainen J (2018) Requirements engineering and continuous deployment. IEEE Softw 35(2):86–90

    Article  Google Scholar 

  46. Zave P, Jackson M (1997) Four dark corners of requirements engineering. ACM Trans Softw Eng Methodol 6(1):1–30

    Article  Google Scholar 

  47. Gunter CA, Gunter EL, Jackson M, Zave P (2000) A reference model for requirements and specifications. IEEE Softw 17(3):37–43

    Article  Google Scholar 

  48. Mahmoud A, Niu N (2015) On the role of semantics in automated requirements tracing. Requir Eng 20(3):281–300

    Article  Google Scholar 

  49. Ezzini S, Abualhaija S, Arora C, Sabetzadeh M, Briand LC (2021) Using domain-specific corpora for improved handling of ambiguity in requirements. In: Proceedings of international conference on software engineering (ICSE), Madrid, Spain, May 2021, pp 1485–1497

  50. Chernov S, Iofciu T, Nejdl W, Zhou X (2006) Extracting semantics relationships between wikipedia categories. In: Proceedings of the workshop on semantic Wikis (SemWiki), Budva, Montenegro

  51. Beautiful S. A Python library for pulling data out of HTML and XML Files. Last accessed on 2022/09/09 16:36:31. [Online]. Available: https://www.crummy.com/software/BeautifulSoup/

  52. Abualhaija S, Arora C, Sabetzadeh M, Briand LC, Vaz E (2019) A machine learning-based approach for demarcating requirements in textual specifications. In: Proceedings of the international requirements engineering conference (RE), Jeju Island, South Korea, September 2019, pp 51–62

  53. Lin X, Peng Z, Niu N, Wang W, Liu H (2021) Finding metamorphic relations for scientific software. In: Proceedings of international conference on software engineering (ICSE) companion volume, Madrid, Spain, May 2021, pp 254–255

  54. spaCy. Industrial-strength natural language processing in Python. Last accessed on 2022/09/09 16:36:31. [Online]. Available: https://spacy.io/

  55. Kübler S, McDonald R, Nivre J (2009) Dependency parsing. Morgan & Claypool Publishers

    Book  Google Scholar 

  56. Dalpiaz F, Dell’Anna D, Aydemir FB, cCevikol S (2019) Requirements classification with interpretable machine learning and dependency parsing. In: Proceedings of the international requirements engineering conference (RE), Jeju Island, South Korea, September 2019, pp 142–152

  57. Cohen KB, Christiansen T, Hunter LE (2011) Parenthetically speaking: classifying the contents of parentheses for text mining. In: Proceedings of the annual symposium on biomedical and health informatics (AMIA), Washington, DC, USA, October 2011, pp 267–272

  58. Klaussner C, Zhekova D (2011) Pattern-based ontology construction from selected Wikipedia pages. In: Proceedings of the international conference on recent advances in natural language processing (RANLP) Student Research Workshop, Hissar, Bulgaria, September 2011, pp 103–108

  59. Hearst MA (1992) Automatic acquisition of hyponyms from large text corpora. In: Proceedings of the international conference on computational linguistics (COLING), Nantes, France, August 1992, pp 539–545

  60. Klaussner C, Zhekova D (2011) Lexico-syntactic patterns for automatic ontology building. In: Proceedings of the international conference on recent advances in natural language processing (RANLP) Student Research Workshop, Hissar, Bulgaria, September 2011, pp 109–114

  61. Finin TW (1980) The semantic interpretation of nominal compounds. In: Proceedings of the annual national conference on artificial intelligence (AAAI), Stanford, CA, USA, August 1980, pp 310–312

  62. Peng Z, Niu N (2021) Co-AI: a Colab-based tool for abstraction identification. In: Proceedings of the international requirements engineering conference (RE), Notre Dame, IN, USA, September 2021, pp 420–421

  63. Liu H, Shen M, Zhu J, Niu N, Li G, Zhang L (2022) Deep learning based program generation from requirements text: are we there yet? IEEE Trans Softw Eng 48(4):1268–1289

    Article  Google Scholar 

  64. Nyamawe AS, Liu H, Niu N, Umer Q, Niu Z (2019) Automated recommendation of software refactorings based on feature requests. In: Proceedings of the international requirements engineering conference (RE), Jeju Island, South Korea, September 2019, pp 187–198

  65. Nyamawe AS, Liu H, Niu N, Umer Q, Niu Z (2018) Recommending refactoring solutions based on traceability and code metrics. IEEE Access 6:49–475

    Article  Google Scholar 

  66. Nyamawe AS, Liu H, Niu N, Umer Q, Niu Z (2020) Feature requests-based recommendation of software refactorings. Empir Softw Eng 25(5):4315–4347

    Article  Google Scholar 

  67. Niu N, Savolainen J, Bhowmik T, Mahmoud A, Reddivari S (2012) A framework for examining topical locality in object-oriented software. In: Proceedings of the annual IEEE computer software and applications conference (COMPSAC), Izmir, Turkey, July 2012, pp 219–224

  68. OpenEMR. A medical practice management software system supporting electronic medical records (EMR). Last accessed on 2022/09/09 16:36:31. [Online]. Available: https://en.wikipedia.org/wiki/OpenEMR

  69. OpenMRS. A collaborative open-source project on medical record systems (MRS). Last accessed on 2022/09/09 16:36:31. [Online]. Available: https://en.wikipedia.org/wiki/OpenMRS

  70. OpenEMR Features. Features of OpenEMR. Last accessed on 2022/09/09 16:36:31. [Online]. Available: https://www.open-emr.org/wiki/index.php/OpenEMR_Features

  71. OpenMRS User Guide. A complete user guide for OpenMRS. Last accessed on 2022/09/09 16:36:31. [Online]. Available: https://wiki.openmrs.org/display/docs/User+Guide

  72. Maier D (1978) The complexity of some problems on subsequences and supersequences. J ACM 25(2):322–336

    Article  MathSciNet  MATH  Google Scholar 

  73. Lin X, Simon M, Peng Z, Niu N (2021) Discovering metamorphic relations for scientific software from user forums. Comput Sci Eng 23(2):65–72

    Article  Google Scholar 

  74. Hsia P, Kung DC, Sell C (1997) Software requirements and acceptance testing. Ann Softw Eng 3:291–317

    Article  Google Scholar 

  75. Haugset B, Hanssen GK (2008) Automated acceptance testing: a literature review and an industrial case study. In: Proceedings of the agile development conference (AGILE), Toronto, ON, Canada, August 2008, pp 27–38

  76. Cisco. What’s new for the latest channel of Webex meetings. Last accessed on 2022/09/09 16:36:31. [Online]. Available: https://help.webex.com/en-US/article/xcwws1/What’s-New-for-the-Latest-Channel-of-Webex-Meetings

  77. Kohavi R, Tang D, Xu Y (2020) Trustworthy online controlled experiments: a practical guide to A/B testing. Cambridge University Press

    Book  Google Scholar 

  78. NLTK Project. Natural language toolkit. Last accessed on 2022/09/09 16:36:31. [Online]. Available: https://www.nltk.org/

  79. Lilian JF, Sundarakantham K, Rajashree H, Shalinie SM (2019) SSE: semantic sentence embedding for learning user interactions. In: Proceedings of the international conference on computing, communication and networking technologies (ICCCNT), Kanpur, India, July 2019, pp 1–5

  80. Killawala A, Khokhlov I, Reznik L (2018) Computational intelligence framework for automatic quiz question generation. In: Proceedings of the international conference on fuzzy systems (FUZZ-IEEE), Rio de Janeiro, Brazil, July 2018, pp 1–8

  81. Segura S, Fraser G, Sánchez AB, Cortés AR (2016) A survey on metamorphic testing. IEEE Trans Softw Eng 42(9):805–824

    Article  Google Scholar 

  82. Chen TY, Kuo F-C, Liu H, Poon P-L, Towey D, Tse TH, Zhou ZQ (2018) Metamorphic testing: a review of challenges and opportunities. ACM Comput Surv 51(1):4:1-4:27

    Google Scholar 

  83. Auer F, Ros R, Kaltenbrunner L, Runeson P, Felderer M (2021) Controlled experimentation in continuous experimentation: knowledge and challenges. Inform Softw Technol 134:106 551:1–106 551:16

  84. Niu N, Koshoffer A, Newman L, Khatwani C, Samarasinghe C, Savolainen J (2016) Advancing repeated research in requirements engineering: a theoretical replication of viewpoint merging. In: Proceedings of the international requirements engineering conference (RE), Beijing, China, September 2016, pp 186–195

  85. Khatwani C, Jin X, Niu N, Koshoffer A, Newman L, Savolainen J (2017) Advancing viewpoint merging in requirements engineering: a theoretical replication and explanatory study. Requir Eng 22(3):317–338

    Article  Google Scholar 

Download references

Acknowledgements

We thank Sarah Sturmer and Sreelekhaa Nagamalli Santhoshkumar from the University of Cincinnati for their preliminary work on related research topics and their insightful comments on this work. The research is partially supported by the National Natural Science Foundation of China under Grant No. 62192731, 61802374, 62002348, and 62072442, the National Key Research and Development Program of China under Grant No. 2018YFB1403400, and Youth Innovation Promotion Association CAS.

Author information

Authors and Affiliations

  1. University of Cincinnati, Cincinnati, OH, USA

    Zedong Peng, Prachi Rathod & Nan Niu

  2. Mississippi State University, Mississippi State, Starkville, MS, USA

    Tanmay Bhowmik

  3. Beijing Institute of Technology, Beijing, China

    Hui Liu

  4. Institute of Software Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China

    Lin Shi

  5. Peking University Beijing, Beijing, China

    Zhi Jin

Authors
  1. Zedong Peng
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Prachi Rathod
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Nan Niu
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Tanmay Bhowmik
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Hui Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Lin Shi
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Zhi Jin
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Nan Niu.

Ethics declarations

Conflicts of interest

The authors have no conflicts of interest to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Peng, Z., Rathod, P., Niu, N. et al. Testing software’s changing features with environment-driven abstraction identification. Requirements Eng 27, 405–427 (2022). https://doi.org/10.1007/s00766-022-00390-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00766-022-00390-8

Keywords