Automated Quality Concerns Extraction from User Stories and Acceptance Criteria for Early Architectural Decisions

Abstract

User stories serve as a fundamental tool in agile software development methodologies, articulating the functional requirements of a system from an end-user perspective. However, while user stories are crucial for capturing the desired features and functionalities, they frequently overlook the non-functional aspects critical to the system’s success. Despite their paramount importance, these quality concerns often remain implicit or underrepresented in user stories, necessitating a deliberate effort to extract them during the elicitation and architectural analysis phases. Failure to address these quality concerns upfront may lead to poor architectural decisions. Consequently, this oversight may result in sub-optimal system designs, increased development costs, delayed time-to-market, diminished user satisfaction, and increased operational risks. This paper presents an ISO-25010 compliant Transfer Learning approach for automated quality concerns extraction from user stories and corresponding acceptance criteria. The proposed solution is constructed upon the Transformer-based RoBERTa-Large model, leveraging and extending its pre-trained capabilities. This approach proficiently classifies user stories and acceptance criteria into 5 most critical user quality concerns including Usability, Performance, Reliability, Security, and Compatibility. This process involves cleaning and preprocessing the dataset followed by fine-tuning the pre-trained models on the refined data set. A comparative analysis of the Three mainstream BERT variants including RoBERTa-base, DistilBERT and XLNET is also provided. Considering the non-availability of public data sets in this scope, a dataset of approximately 1000 user stories with acceptance criteria was compiled from diverse sources and real-world projects. This dataset was subsequently labeled through an extensive labeling activity. The findings suggest that the RoBERTa-Large fine-tuned variant achieves an impressive level of performance in terms of accuracy, precision, recall and Avg F1 score.