Skip to main content
Springer Nature Link
Log in

Exploring Classification Consistency of Natural Language Requirements Using GPT-4o

  • Conference paper
  • First Online:
Software Business (ICSOB 2024)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 539))

Included in the following conference series:

  • 173 Accesses

Abstract

Classifying natural language requirements (NLRs) is challenging, especially with large volumes. Research shows that Large Language Models can assist by categorizing NLRs into functional requirements (FR) and non-functional requirements (NFRs). However, Generative Pretrained Transformer (GPT) models are not typically favored for this task due to concerns about consistency. This paper investigates the consistency when a GPT model classifies NLRs into FRs and NFRs using a zero-shot learning approach. Results show that ChatGPT-4o performs better for FRs, a temperature parameter set to 1 yields the highest consistency, while NFR classification improves with higher temperatures.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Kassab, M.: State of practice in requirements engineering: contemporary data. Innov. Syst. Softw. Eng. 10, 235–241 (2014)

    MATH  Google Scholar 

  2. Abad, Z.S.H., Karras, O., Ghazi, P., Glinz, M., Ruhe, G., Schneider, K.: What works better? A study of classifying requirements. In: 2017 IEEE 25th International Requirements Engineering Conference, pp. 496–501. IEEE (2017)

    Google Scholar 

  3. Bashir, S., Abbas, M., Ferrari, A., Saadatmand, M., Lindberg, P.: Requirements classification for smart allocation: a case study in the railway industry. In: 2023 IEEE 31st International Requirements Engineering Conference (RE), pp. 201–211. IEEE (2023)

    Google Scholar 

  4. Hey, T., Keim, J., Koziolek, A., Tichy, W.F.: NoRBERT: transfer learning for requirements classification. In: 2020 IEEE 28th International Requirements Engineering Conference (RE), pp. 169–179. IEEE (2020)

    Google Scholar 

  5. Kurtanović, Z., Maalej, W.: Automatically classifying functional and non-functional requirements using supervised machine learning. In: 2017 IEEE 25th International Requirements Engineering Conference (RE), pp. 490–495. IEEE (2017)

    Google Scholar 

  6. Wang, W., Zheng, V.W., Yu, H., Miao, C.: A survey of zero-shot learning: settings, methods, and applications. ACM Trans. Intell. Syst. Technol. 10, 1–37 (2019)

    MATH  Google Scholar 

  7. Alhoshan, W., Ferrari, A., Zhao, L.: Zero-shot learning for requirements classification: an exploratory study. Inf. Softw. Technol. 159, 107202 (2023)

    Google Scholar 

  8. Ouyang, S., Zhang, J.M., Harman, M., Wang, M.: LLM is like a box of chocolates: the non-determinism of ChatGPT in code generation (2023). arXiv:2308.02828

  9. Chen, B., et al.: On the use of GPT-4 for creating goal models: an exploratory study. In: 31st International Requirements Engineering Conference Workshops (REW 2023), pp. 262–271. IEEE (2023)

    Google Scholar 

  10. Larochelle, H., Erham, D., Bengio, Y.: Zero-data learning of new tasks. In: Cohn, A. (ed.) AAAI’08: Proceedings of the 23rd National Conference on Artificial Intelligence, vol. Volume 2, pp. 646–651. ACM (2008)

    Google Scholar 

  11. Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) 36th Conference on Neural Information Processing Systems (NeurIPS 2022), pp. 22199–22213. ACM (2022)

    Google Scholar 

  12. Kaur, K., Kaur, P.: The application of AI techniques in requirements classification: a systematic mapping. Artif. Intell. Rev. 57, 57 (2024)

    MATH  Google Scholar 

  13. Winkler, J., Vogelsang, A.: Automatic classification of requirements based on convolutional neural networks. In: 2016 IEEE 24th International Requirements Engineering Conference Workshops, pp. 39–45. IEEE, Beijing, China (2016)

    Google Scholar 

  14. Cleland-Huang, J., Settimi, R., Zou, X., Solc, P.: Automated classification of non-functional requirements. Requirements Eng. 12, 103–120 (2007)

    MATH  Google Scholar 

  15. Goutte, C., Gaussier, E.: A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In: Losada, D.E., Fernández-Luna, J.M. (eds.) Advances in Information Retrieval, ECIR 2005. Lecture Notes in Computer Science, vol. 3408, pp. 345–359. Springer, Berlin, Heidelberg (2005). https://doi.org/10.1007/978-3-540-31865-1_25

  16. Fleiss, J.L., Levin, B., Paik, M.C.: Statistical Methods for Rates and Proportions. Wiley, Hoboken (2013)

    Google Scholar 

  17. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Egnell, B., Wesslen, A.: Experimentation in Software Engineering: An Introduction. Kluwer Academic Publishers, Boston (2000)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Informatics, Örebro University, 701 82, Örebro, Sweden

    Fredrik Karlsson, Panagiota Chatzipetrou, Shang Gao & Tanja Elina Havstorm

Authors
  1. Fredrik Karlsson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Panagiota Chatzipetrou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shang Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tanja Elina Havstorm
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fredrik Karlsson .

Editor information

Editors and Affiliations

  1. RISE Research Institutes of Sweden, Gothenburg, Sweden

    Efi Papatheocharous

  2. Wageningen University and Research, Wageningen, The Netherlands

    Siamak Farshidi

  3. Universiteit Utrecht, Utrecht, The Netherlands

    Slinger Jansen

  4. LUT University, Lahti, Finland

    Sonja Hyrynsalmi

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Karlsson, F., Chatzipetrou, P., Gao, S., Havstorm, T.E. (2025). Exploring Classification Consistency of Natural Language Requirements Using GPT-4o. In: Papatheocharous, E., Farshidi, S., Jansen, S., Hyrynsalmi, S. (eds) Software Business. ICSOB 2024. Lecture Notes in Business Information Processing, vol 539. Springer, Cham. https://doi.org/10.1007/978-3-031-85849-9_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-85849-9_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-85848-2

  • Online ISBN: 978-3-031-85849-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics