Skip to main content
SpringerLink
Log in

An NLP approach for cross-domain ambiguity detection in requirements engineering

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

During requirements elicitation, different stakeholders with diverse backgrounds and skills need to effectively communicate to reach a shared understanding of the problem at hand. Linguistic ambiguity due to terminological discrepancies may occur between stakeholders that belong to different technical domains. If not properly addressed, ambiguity can create frustration and distrust during requirements elicitation meetings, and lead to problems at later stages of development. This paper presents a natural language processing approach to identify ambiguous terms between different domains, and rank them by ambiguity score. The approach is based on building domain-specific language models, one for each stakeholders’ domain. Word embeddings from each language model are compared in order to measure the differences of use of a term, thus estimating its potential ambiguity across the domains of interest. We evaluate the approach on seven potential elicitation scenarios involving five domains. In the evaluation, we compare the ambiguity rankings automatically produced with the ones manually obtained by the authors as well as by multiple annotators recruited through Amazon Mechanical Turk. The rankings produced by the approach lead to a maximum Kendall’s Tau of 88%. However, for several elicitation scenarios, the application of the approach was unsuccessful in terms of performance. Analysis of the agreement among annotators and of the observed inaccuracies offer hints for further research on the relationship between domain knowledge and natural language ambiguity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. https://doi.org/10.5281/zenodo.1476902.

  2. https://en.wikipedia.org/wiki/Category:Computer_science.

  3. In the pseudo-code, array indexes start from 1.

  4. When n = 2, the variance is equivalent to the mean squared error.

  5. https://pypi.org/project/wikipedia/.

  6. https://radimrehurek.com/gensim/.

  7. https://spacy.io.

  8. spaCy is used also to identify nouns through POS tagging. The authors of spaCy (Honnibal and Montani 2017) report a 97% accuracy for the pretrained POS tagger for English, which is in line with the state of the art (Manning 2011). Given the high accuracy, we consider the influence of POS tagger errors as negligible.

  9. https://doi.org/10.5281/zenodo.1476902.

  10. In this preliminary task, the authors annotated a group of 60 sentences for the (I3) Medical Software (CS, MED) and 60 sentences for the (M2) Medical Robot scenario (CS, EEN, MEN, MED)—the sentence sets were not included in the evaluation presented in this work. The annotation was followed by a discussion on the annotated data and evident cases of disagreement, which allowed the authors to have an initial common ground to select the examples.

  11. The original Kendall’s Tau measure does not provide a policy to handle ties.

  12. https://arxiv.org.

  13. https://www.qualicen.de/en/.

  14. https://qracorp.com.

  15. https://wordnet.princeton.edu.

  16. Preliminary experiments were performed in this direction, to obtain the currently adopted parameters’ values. However, we reckon that a systematic evaluation campaign is required, as there may be an optimal parameters’ selection for each domain group.

References

  • Agirre, E., Edmonds, P.: Word Sense Disambiguation: Algorithms and Applications, vol. 33. Springer, Berlin (2007). https://doi.org/10.1007/978-1-4020-4809-8

  • Agresti, A.: Analysis of Ordinal Categorical Data, vol. 656. Wiley, Hoboken (2010)

    Book  MATH  Google Scholar 

  • Ambriola, V., Gervasi, V.: On the systematic analysis of natural language requirements with CIRCE. ASE 13(1), 107–167 (2006). https://doi.org/10.1007/s10515-006-5468-2

  • Arora, C., Sabetzadeh, M., Briand, L., Zimmer, F.: Automated checking of conformance to requirements templates using natural language processing. TSE 41(10), 944–968 (2015). https://doi.org/10.1109/TSE.2015.2428709

    Article  Google Scholar 

  • Avramidis, E.: Rankeval: open tool for evaluation of machine-learned ranking. Prague Bull. Math. Linguist. 100, 63–72 (2013). https://doi.org/10.2478/pralin-2013-0012

    Article  Google Scholar 

  • Berry, D.M., Kamsties, E.: Ambiguity in requirements specification. In: do Prado Leite, J.C.S., Doorn, J.H. (eds) Perspectives on Software Requirements. The Springer International Series in Engineering and Computer Science, vol. 753, pp. 7–44. Springer, Boston, MA (2004). https://doi.org/10.1007/978-1-4615-0465-8_2

  • Berry, D.M., Kamsties, E.: The syntactically dangerous all and plural in specifications. IEEE Softw. 22(1), 55–57 (2005). https://doi.org/10.1109/MS.2005.22

    Article  Google Scholar 

  • Berry, D.M., Kamsties, E., Krieger, M.M.: From contract drafting to software specification: linguistic sources of ambiguity. https://cs.uwaterloo.ca/~dberry/handbook/ambiguityHandbook.pdf (2003). Accessed 10 June 2019

  • Blei, D.M.: Probabilistic topic models. Commun. ACM 55(4), 77–84 (2012). https://doi.org/10.1145/2133806.2133826

    Article  Google Scholar 

  • Camacho-Collados, J., Pilehvar, T.: From word to sense embeddings: a survey on vector representations of meaning. arXiv preprint arXiv:1805.04032 (2018)

  • Casamayor, A., Godoy, D., Campo, M.: Functional grouping of natural language requirements for assistance in architectural software design. KBS 30, 78–86 (2012). https://doi.org/10.1016/j.knosys.2011.12.009

    Article  Google Scholar 

  • Chantree, F., Nuseibeh, B., De Roeck, A.N., Willis, A.: Identifying nocuous ambiguities in natural language requirements. In: RE’06, pp. 56–65 (2006). https://doi.org/10.1109/RE.2006.31

  • Chen, X., Liu, Z., Sun, M.: A unified model for word sense representation and disambiguation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1025–1035 (2014). https://doi.org/10.3115/v1/D14-1110

  • Cleland-Huang, J.: Mining domain knowledge [requirements]. IEEE Softw. 32(3), 16–19 (2015). https://doi.org/10.1109/MS.2015.67

    Article  Google Scholar 

  • Nattoch Dag, J., Gervasi, V., Brinkkemper, S., et al.: A linguistic-engineering approach to large-scale requirements management. IEEE Softw. 1, 32–39 (2005). https://doi.org/10.1109/MS.2005.1

    Article  Google Scholar 

  • Dalpiaz, F., van der Schalk, I., Lucassen, G.: Pinpointing ambiguity and incompleteness in requirements engineering via information visualization and NLP. In: International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, pp. 119–135 (2018). https://doi.org/10.1007/978-3-319-77243-1_8

  • Dieste, O., Juristo, N.: Systematic review and aggregation of empirical studies on elicitation techniques. IEEE Trans. Softw. Eng. 37(2), 283–304 (2011). https://doi.org/10.1109/TSE.2010.33

    Article  Google Scholar 

  • Evans, M.C., Bhatia, J., Wadkar, S., Breaux, T.D.: An evaluation of constituency-based hyponymy extraction from privacy policies. In: Requirements Engineering Conference (RE), 2017 IEEE 25th International. IEEE, pp. 312–321 (2017). https://doi.org/10.1109/RE.2017.87

  • Falessi, D., Cantone, G., Canfora, G.: Empirical principles and an industrial case study in retrieving equivalent requirements via natural language processing techniques. IEEE Trans. Softw. Eng. 39(1), 18–44 (2013). https://doi.org/10.1109/TSE.2011.122

    Article  Google Scholar 

  • Femmer, H., Kučera, J., Vetrò, A.: On the impact of passive voice requirements on domain modelling. In: ESEM. ACM, p. 21 (2014). https://doi.org/10.1145/2652524.2652554

  • Femmer, H., Fernández, D.M., Wagner, S., Eder, S.: Rapid quality assurance with requirements smells. JSS 123, 190–213 (2017). https://doi.org/10.1016/j.jss.2016.02.047

    Article  Google Scholar 

  • Fernández, D.M., Wagner, S., Kalinowski, M., Felderer, M., Mafra, P., Vetrò, A., Conte, T., Christiansson, M.T., Greer, D., Lassenius, C., Männistö, T., Nayabi, M., Oivo, M., Penzenstadler, B., Pfahl, D., Prikladnicki, R., Ruhe, G., Schekelmann, A., Sen, S., Spinola, R., Tuzcu, A., de la Vara, J.L., Wieringa, R.: Naming the pain in requirements engineering. Empir. Softw. Eng. 22(5), 2298–2338 (2017). https://doi.org/10.1007/s10664-016-9451-7

    Article  Google Scholar 

  • Ferrari, A., Gnesi, S.: Using collective intelligence to detect pragmatic ambiguities. In: RE’12. IEEE, pp. 191–200 (2012). https://doi.org/10.1109/RE.2012.6345803

  • Ferrari, A., Spoletini, P., Gnesi, S.: Ambiguity and tacit knowledge in requirements elicitation interviews. REJ 21(3), 333–355 (2016). https://doi.org/10.1007/s00766-016-0249-3

    Article  Google Scholar 

  • Ferrari, A., Dell’Orletta, F., Esuli, A., Gervasi, V., Gnesi, S.: Natural language requirements processing: a 4D vision. IEEE Softw. 6, 28–35 (2017a). https://doi.org/10.1109/MS.2017.4121207

    Article  Google Scholar 

  • Ferrari, A., Donati, B., Gnesi, S.: Detecting domain-specific ambiguities: an NLP approach based on Wikipedia crawling and word embeddings. In: 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW). IEEE, pp. 393–399 (2017b). https://doi.org/10.1109/REW.2017.20

  • Ferrari, A., Esuli, A., Gnesi, S.: Identification of cross-domain ambiguity with language models. In: Groen, E.C., Harrison, R., Murukannaiah, P.K., Vogelsang, A. (eds) 5th International Workshop on Artificial Intelligence for Requirements Engineering, AIRE@RE 2018, Banff, AB, Canada, 21 Aug 2018. IEEE, pp. 31–38 (2018a). https://doi.org/10.1109/AIRE.2018.00011

  • Ferrari, A., Gori, G., Rosadini, B., Trotta, I., Bacherini, S., Fantechi, A., Gnesi, S.: Detecting requirements defects with NLP patterns: an industrial experience in the railway domain. Empir. Softw. Eng. 23(6), 3684–3733 (2018b). https://doi.org/10.1007/s10664-018-9596-7

    Article  Google Scholar 

  • Firth, J.R.: Selected Papers of JR Firth, 1952–59. Indiana University Press, Bloomington (1968)

    Google Scholar 

  • Flekova, L., Gurevych, I.: Supersense embeddings: a unified model for supersense interpretation, prediction, and utilization. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, pp. 2029–2041 (2016). https://doi.org/10.18653/v1/P16-1191

  • Gacitua, R., Sawyer, P., Gervasi, V.: On the effectiveness of abstraction identification in requirements engineering. In: Proceedings of the 18th IEEE International Requirements Engineering Conference (RE’10). IEEE, pp. 5–14 (2010). https://doi.org/10.1109/RE.2010.12

  • Gleich, B., Creighton, O., Kof, L.: Ambiguity detection: Towards a tool explaining ambiguity sources. In: REFSQ’10, vol. 6182. Springer, LNCS, pp. 218–232 (2010). https://doi.org/10.1007/978-3-642-14192-8_20

  • Guo, J., Cheng, J., Cleland-Huang, J.: Semantically enhanced software traceability using deep learning techniques. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, pp. 3–14 (2017). https://doi.org/10.1109/ICSE.2017.9

  • Harris, Z.S.: Distributional structure. Word 10(2–3), 146–162 (1954). https://doi.org/10.1080/00437956.1954.11659520

    Article  Google Scholar 

  • Hoffman, R.R., Shadbolt, N.R., Burton, A.M., Klein, G.: Eliciting knowledge from experts: a methodological analysis. Organ. Behav. Hum. Decis. Process. 62(2), 129–158 (1995). https://doi.org/10.1006/obhd.1995.1039

    Article  Google Scholar 

  • Honnibal, M., Montani, I.: spaCy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing (2017) (to appear)

  • Jha, N., Mahmoud, A.: Mining user requirements from application store reviews using frame semantics. In: International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, pp. 273–287 (2017). https://doi.org/10.1007/978-3-319-54045-0_20

  • Kendall, M.G.: The treatment of ties in ranking problems. Biometrika 33(3), 239–251 (1945). https://doi.org/10.2307/2332303

    Article  MathSciNet  MATH  Google Scholar 

  • Kof, L.: From requirements documents to system models: a tool for interactive semi-automatic translation. In: 2010 18th IEEE International Requirements Engineering Conference, pp. 391–392 (2010). https://doi.org/10.1109/RE.2010.53

  • Krisch, J., Houdek, F.: The myth of bad passive voice and weak words an empirical investigation in the automotive industry. In: 2015 IEEE 23rd International Requirements Engineering Conference (RE). IEEE, pp. 344–351 (2015). https://doi.org/10.1109/RE.2015.7320451

  • Lami, G., Gnesi, S., Fabbrini, F., Fusani, M.: The linguistic approach to the natural language requirements quality: Benefit of the use of an automatic tool. In: Proceedings 26th Annual NASA Goddard Software Engineering Workshop(SEW), vol. 00, p. 97 (2001). https://doi.org/10.1109/SEW.2001.992662

  • Landis, J.R., Koch, G.G.: An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 33(2), 363–374 (1977). https://doi.org/10.2307/2529786

    Article  MATH  Google Scholar 

  • Lee, Y.K., Ng, H.T.: An empirical evaluation of knowledge sources and learning algorithms for word sense disambiguation. In: Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing-Volume 10. Association for Computational Linguistics, pp. 41–48 (2002). https://doi.org/10.3115/1118693.1118699

  • Levy, O., Goldberg, Y.: Neural word embedding as implicit matrix factorization. In: Advances in Neural Information Processing Systems, pp. 2177–2185. http://papers.nips.cc/paper/5477-neuralword-embedding-as-implicit-matrix-factorization.pdf (2014). Accessed 10 June 2019

  • Lian, X., Rahimi, M., Cleland-Huang, J., Zhang, L., Ferrai, R., Smith, M.: Mining requirements knowledge from collections of domain documents. In: 2016 IEEE 24th International Requirements Engineering Conference (RE). IEEE, pp. 156–165 (2016). https://doi.org/10.1109/RE.2016.50

  • Maalej, W., Nabil, H.: Bug report, feature request, or simply praise? on automatically classifying app reviews. In: Proceedings of the 23rd IEEE International Requirements Engineering Conference, (RE’15). IEEE, pp. 116–125 (2015). https://doi.org/10.1109/RE.2015.7320414

  • Manning, C.D.: Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In: International Conference on Intelligent Text Processing and Computational Linguistics. Springer, pp. 171–189 (2011)

  • Massey, A.K., Rutledge, R.L., Anton, A.I., Swire, P.P.: Identifying and classifying ambiguity for regulatory requirements. In: RE’14. IEEE, pp. 83–92 (2014). https://doi.org/10.1109/RE.2014.6912250

  • Mavin, A., Wilkinson, P., Harwood, A., Novak, M.: Easy approach to requirements syntax (EARS). In: RE’09. IEEE, pp. 317–322 (2009). https://doi.org/10.1109/RE.2009.9

  • Mich, L.: NL-OOPS: from natural language to object oriented requirements using the natural language processing system LOLITA. NLE 2(2), 161–187 (1996). https://doi.org/10.1017/S1351324996001337

    Article  Google Scholar 

  • Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013). arXiv:1310.4546

  • Miller, G.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998)

    MATH  Google Scholar 

  • Navigli, R.: Word sense disambiguation: A survey. ACM Comput. Surv. (CSUR) 41(2), 10 (2009). https://doi.org/10.1145/1459352.1459355

    Article  Google Scholar 

  • Pedersen, T.: A simple approach to building ensembles of naive bayesian classifiers for word sense disambiguation. In: Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, pp. 63–69. Association for Computational Linguistics. http://www.aclweb.org/anthology/A00-2009 (2000). Accessed 10 June 2019

  • Pedersen, T.: Unsupervised corpus-based methods for WSD. In: Agirre, E., Edmonds, P. (eds) Word Sense Disambiguation. Text, Speech and Language Technology, vol. 33, pp. 133–166. Springer, Dordrecht (2007). https://doi.org/10.1007/978-1-4020-4809-8_6

  • Pohl, K., Rupp, C.: Requirements Engineering Fundamentals. Rocky Nook Inc, San Rafael (2011)

    Google Scholar 

  • Quirchmayr, T., Paech, B., Kohl, R., Karey, H.: Semi-automatic software feature-relevant information extraction from natural language user manuals. In: Proceedings of the 23rd International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ’17), pp. 255–272, Springer (2017). https://doi.org/10.1007/978-3-319-54045-0_19

  • Raganato, A., Bovi, C.D., Navigli, R.: Neural sequence learning models for word sense disambiguation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1156–1167 (2017). https://doi.org/10.18653/v1/D17-1120

  • Robeer, M., Lucassen, G., van der Werf, J.M.E., Dalpiaz, F., Brinkkemper, S.: Automated extraction of conceptual models from user stories via nlp. In: Proceedings of the 24th IEEE International Requirements Engineering Conference (RE’16), pp. 196–205. IEEE (2016). https://doi.org/10.1109/RE.2016.40

  • Rodriguez, D.V., Carver, D.L., Mahmoud, A.: An efficient wikipedia-based approach for better understanding of natural language text related to user requirements. In: 2018 IEEE Aerospace Conference, pp. 1–16. IEEE (2018) https://doi.org/10.1109/AERO.2018.8396645

  • Rosadini, B., Ferrari, A., Gori, G., Fantechi, A., Gnesi, S., Trotta, I., Bacherini, S.: Using NLP to detect requirements defects: an industrial experience in the railway domain. In: REFSQ, pp. 344–360. Springer (2017) https://doi.org/10.1007/978-3-319-54045-0_24

  • Sleimi, A., Sannier, N., Sabetzadeh, M., Briand, L., Dann, J.: Automated extraction of semantic legal metadata using natural language processing. In: 2018 IEEE 26th International Requirements Engineering Conference (RE), pp. 124–135. IEEE (2018). https://doi.org/10.1109/RE.2018.00022

  • Sultanov, H., Hayes, J.H.: Application of reinforcement learning to requirements engineering: requirements tracing. In: Proceedings of the 21st IEEE International Requirements Engineering Conference (RE’13), pp. 52–61. IEEE (2013). https://doi.org/10.1109/RE.2013.6636705

  • Taghipour, K., Ng, H.T.: Semi-supervised word sense disambiguation using word embeddings in general and specific domains. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 314–323 (2015) https://doi.org/10.3115/v1/N15-1035

  • Tjong, S., Berry, D.: The design of SREE a prototype potential ambiguity finder for requirements specifications and lessons learned. In: REFSQ’13, LNCS, vol. 7830, pp. 80–95 (2013). https://doi.org/10.1007/978-3-642-37422-7_6

  • Wang, W., Niu, N., Liu, H., Niu, Z.: Enhancing automated requirements traceability by resolving polysemy. In: 2018 IEEE 26th International Requirements Engineering Conference (RE), pp. 40–51. IEEE (2018). https://doi.org/10.1109/RE.2018.00-53

  • Yang, H., De Roeck, A.N., Gervasi, V., Willis, A., Nuseibeh, B.: Analysing anaphoric ambiguity in natural language requirements. Requir. Eng. 16(3), 163–189 (2011). https://doi.org/10.1007/s00766-011-0119-y

    Article  Google Scholar 

  • Yuan, D., Richardson, J., Doherty, R., Evans, C., Altendorf, E.: Semi-supervised word sense disambiguation with neural models. arXiv preprint arXiv:1603.07012 (2016)

  • Yue, T., Briand, L.C., Labiche, Y.: aToucan: an automated framework to derive UML analysis models from use case models. ACM Trans. Softw. Eng. Methodol. (TOSEM) 24(3), 13 (2015). https://doi.org/10.1145/2699697

    Article  Google Scholar 

  • Zowghi, D., Coulin, C.: Requirements elicitation: a survey of techniques, approaches, and tools. In: Engineering and Managing Software Requirements, pp. 19–46. Springer (2005). https://doi.org/10.1007/3-540-28244-0_2

Download references

Author information

Authors and Affiliations

  1. CNR-ISTI, Via G. Moruzzi 1, 56124, Pisa, Italy

    Alessio Ferrari & Andrea Esuli

Authors
  1. Alessio Ferrari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrea Esuli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alessio Ferrari.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ferrari, A., Esuli, A. An NLP approach for cross-domain ambiguity detection in requirements engineering. Autom Softw Eng 26, 559–598 (2019). https://doi.org/10.1007/s10515-019-00261-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-019-00261-7

Keywords

Search

Navigation