Abstract:
Software Maintenance and Evolution (SME) is moving fast with the assistance of artificial intelligence (AI), especially Large Language Models (LLM). Researchers have alre...Show MoreMetadata
Abstract:
Software Maintenance and Evolution (SME) is moving fast with the assistance of artificial intelligence (AI), especially Large Language Models (LLM). Researchers have already started automating various activities of the SME workflow. Un-derstanding the requirements for maintenance and development work i.e. Requirements Engineering (RE) is a crucial phase that kicks off the SME workflow through multiple discussions on a proposed scope of work documented in different forms. The RE phase ends with a list of user stories for each unit task and usually created and tracked on a project management tool such as GitHub, Jira, AzurDev, etc. In this research, we collaborated with Bell Mobility to develop a tool “Geneus” (Generate UserSory) using GPT-4-turbo to automatically create user stories from software requirements documents. Requirements documents are usually long and contain complex information. Since LLMs typically suffer from hallucination when the input is too complex, this paper proposes a new prompting strategy, “Refine and Thought” (RaT), to mitigate that issue and improve the performance of the LLM in prompts with large and noisy contexts. Along with manual evaluation using RUST (Readability, Understandability, Specificity, Technical-aspects) survey questionnaire, automatic evaluation with BERTScore, and AlignScore evaluation metrics are used to evaluate the results of the “Geneus” tool. Results show that our method with RaT performs consistently better in most of the cases of interactions compared to the single-shot baseline method. However, the BERTScore and AlignScore test results are not consistent. In the median case, Geneus performs significantly better in all three interactions (requirements specifi-cation, user story details, and test case specifications) according to AlignScorebut it shows slightly low performance in requirements specifications according to BERTScore. Distilling RE documents requires significant time & effort from the senior members of th...
Date of Conference: 06-11 October 2024
Date Added to IEEE Xplore: 19 December 2024
ISBN Information: