Authors:
Lucas Liebe
1
;
Jannis Baum
1
;
Tilman Schütze
1
;
Tim Cech
2
;
Willy Scheibel
1
and
Jürgen Döllner
1
Affiliations:
1
University of Potsdam, Digital Engineering Faculty, Hasso Plattner Institute, Germany
;
2
University of Potsdam, Digital Engineering Faculty, Germany
Keyword(s):
Explainable AI, Text Generation, Linguistic Text Analysis, Topic Modeling, Entity Recognition, Stylometry.
Abstract:
Text synthesis tools are becoming increasingly popular and better at mimicking human language. In trust-sensitive decisions, such as plagiarism and fraud detection, identifying AI-generated texts poses larger difficulties: decisions need to be made explainable to ensure trust and accountability. To support users in identifying AI-generated texts, we propose the tool UNCOVER. The tool analyses texts through three explainable linguistic approaches: Stylometric writing style analysis, topic modeling, and entity recognition. The result of the tool is a prediction and visualization of the analysis. We evaluate the tool on news articles by means of accuracy of the prediction and an expert study with 13 participants. The final prediction is based on classification of stylometric and evolving topic analysis. It achieved an accuracy of 70.4% and a weighted F1-score of 85.6%. The participants preferred to base their assessment on the prediction and the topic graph. In contrast, they found the
entity recognition to be an ineffective indicator. Moreover, five participants highlighted the explainable aspects of UNCOVER and overall the participants achieved 69% accuracy. Eight participants expressed interest to continue using UNCOVER for identifying AI-generated texts.
(More)