An Ensemble-Based Approach for Generative Language Model Attribution

Abstract

Recently, Large Language Models (LLMs) have gained considerable attention due to their incredible ability to automatically generate texts that closely resemble human-written text. They have become invaluable tools in handling various text-based tasks such as content creation and report generation. Nevertheless, the proliferation of these tools can create undesirable consequences such as generation of false information and plagiarism. A variety of LLMs have been operationalized in the last few years whose abilities are heavily influenced by the quality of their training corpus, model architecture, pre-training tasks, and fine-tuning processes. Our ability to attribute the generated text to a specific LLM will not only help us understand differences in the LLMs’ output characteristics, but also effectively distinguish machine-generated text from human-generated text. In this paper, we study whether a machine learning model can be effectively trained to attribute text to the underlying LLM that generated it. We propose an ensemble neural model that generates probabilities from multiple pre-trained LLMs, which are then used as features for a traditional machine learning classifier. The proposed approach is tested on Automated Text Identification (AuTexTification) datasets in English and Spanish languages. We find that our models outperform various baselines, achieving macro $F_{macro}$ scores of 0.63 and 0.65 for English and Spanish texts, respectively.