A Transformer-Based Generative AI Model in Education: Fine-Tuning BERT for Domain-Specific in Student Advising

Abstract

The Transformer model has inspired state-of-the-art generative NLP models such as the Bidirectional Encoder Representations from Transformers (BERT), Generative Pre-trained Transformer (GPT) and their variations models. Despite the fact that these training methods are responsible to have an effective language model, but the computational cost will be very expensive for performing any NLP tasks. Accordingly, the fine-tuning plays an essential role in improving the performance during the training process and reducing the computational cost. In this study we applied one of the pre-trained model from HuggingFace platform called a BERT-base-uncased model, which it’s variant of BERT and we utilized a specific purpose dataset for high school advising. However, the data is collected from high school and universities websites as well as from educational experts. The data includes enquiries and answers about advising high school students toward their future. This transformer, takes the input as a pair from the context and the question, and the output defined with the start and end positions of the answer in the context. Accordingly, the collected dataset is converted into json file, and then we applied the PyTorch libraries for building both, training and inference models. The ROUGE metrics revealed that the model achieves a good performance in answering to the students’ questions.