Investigating BERT Layer Performance and SMOTE Through MLP-Driven Ablation on Gittercom

Abstract

For software development teams, communication is necessary to preserve growth consciousness, streamline the management of projects, and avoid misunderstandings. Among the features that chat rooms offer to help and satisfy the interaction requirements of software-development groups are instant personal messaging, group conversations, and code knowledge exchange. This is all capable of occurring in real time. As a result, developers are increasingly using chat rooms. One of these prominent forums is Gitter, and the chats it includes might be a goldmine of information for researchers studying open-source software systems. The GitterCom dataset, the biggest repository of Gitter developer messages that have been meticulously labeled and organized, was used in this study to conduct a multi-label categorization for the dataset’s Purpose Category. 9 MLP machine learning classifiers, six feature selection methods, and the layered architecture of the BERT transformer are all subjected to thorough empirical research and evaluation. As a consequence, our research process shows competent results with a maximum AUC score of 0.97 with MLP variants using Adam optimizer(MLP2 and MLP3). Additionally, the research process might be used to text data from software development forums for general multi-label text categorization. The insights of the research, which give a holistic understanding of the Machine learning pipeline driven by BERT, shall serve the research community for preferential selection of Feature selection techniques, BERT layers, and classification model selection, among others for text classification.