Authors:
Viktor Csuvik
1
;
Tibor Gyimóthy
1
and
László Vidács
1
;
2
Affiliations:
1
University of Szeged, Department of Software Engineering, Hungary
;
2
University of Szeged, MTA-SZTE Research Group on Artificial Intelligence, Hungary
Keyword(s):
Automated Program Repair, Transformers, ChatGPT, JavaScript, Java.
Abstract:
ChatGPT, a large language model (LLM) developed by OpenAI, fine-tuned on a massive dataset of text and source code, has recently gained significant attention on the internet. The model, built using the Transformer architecture, is capable of generating human-like text in a variety of tasks. In this paper, we explore the use of ChatGPT for Automated Program Repair (APR); that is, we ask the model to generate repair suggestions for instances of buggy code. We evaluate the effectiveness of our approach by comparing the repair suggestions to those made by human developers. Our results show that ChatGPT is able to generate fixes that are on par with those made by humans. Choosing the right prompt is a key aspect: on average, it was able to propose corrections in 19% of cases, but choosing the wrong input format can drop the performance to as low as 6%. By sampling real-world bugs from seminal APR datasets, generating 1000 input examples for the model, and evaluating the output manually, o
ur study demonstrates the potential of language models for Automated Program Repair and highlights the need for further research in this area.
(More)