Retrieval Augmented Large Language Model Chatbots in Higher Education: A Study on University Open Days

Abstract

This study explores the implementation of Retrieval Augmented Large Language Models (LLMs) toward enhancing prospective student engagement during University open days. This work proposes the use of local, 4-bit quantised LLMs such as Microsoft Phi3, Meta’s LLaMa3, and Mistral AI’s Mistral to facilitate interactive dialogue about the Department of Computer Science at Nottingham Trent University. The proposed approaches are validated through the use of synthetic data generation via the RAGAS framework with additional expert human-in-the-loop oversight. We argue that the current state of the art which often involves ChatGPT as a sole validator is problematic, and we propose the use of an ensemble of multiple local validators that operate when a quorum is present to increase robustness. The results indicate that, while the chatbots are successful in providing the correct information, refining data relevance remains an open issue. Mistral demonstrated the highest performance in terms of information accuracy and coherence of responses, however, it was also the slowest at generating responses.