research-article

How Do Software Developers Use ChatGPT? An Exploratory Study on GitHub Pull Requests

Authors:

Moataz Chouchen,

Narjes Bessghaier,

Mohamed Wiem MkaouerAuthors Info & Claims

MSR '24: Proceedings of the 21st International Conference on Mining Software Repositories

Pages 212 - 216

https://doi.org/10.1145/3643991.3645084

Published: 02 July 2024 Publication History

Abstract

Nowadays, Large Language Models (LLMs) play a pivotal role in software engineering. Developers can use LLMs to address software development-related tasks such as documentation, code refactoring, debugging, and testing. ChatGPT, released by OpenAI, has become the most prominent LLM. In particular, ChatGPT is a cutting-edge tool for providing recommendations and solutions for developers in their pull requests (PRs). However, little is known about the characteristics of PRs that incorporate ChatGPT compared to those without it and what developers usually use it for. To this end, we quantitatively analyzed 243 PRs that listed at least one ChatGPT prompt against a representative sample of 384 PRs without any ChatGPT prompts. Our findings show that developers use ChatGPT in larger, time-consuming pull requests that are five times slower to be closed than PRs that do not use ChatGPT. Furthermore, we perform a qualitative analysis to build a taxonomy of the topics developers primarily address in their prompts. Our analysis results in a taxonomy comprising 8 topics and 32 sub-topics. Our findings highlight that ChatGPT is often used in review-intensive pull requests. Moreover, our taxonomy enriches our understanding of the developer's current applications of ChatGPT.

References

[1]

_____. 2023. The reolication package for our study: https://figshare.com/s/2bc2c11e694997262a87.

[2]

2023. ChatGPT Shared Links FAQ. https://help.openai.com/en/articles/7925741-chatgpt-shared-links-faq. Accessed on: Dec 01, 2023.

[3]

2023. Pull Request #1. https://github.com/Wissididom/Bonbon-Reminder-StreamElements/pull/1 Accessed on: Dec 01, 2023.

[4]

2023. Pull Request #1775. https://github.com/ggerganov/llama.cpp/pull/1775 Accessed on: Dec 01, 2023.

[5]

2023. Pull Request #2. https://github.com/DogeisCut/extensions/pull/2 Accessed on: Dec 01, 2023.

[6]

2023. Pull Request #209. https://github.com/sCrypt-Inc/scryptTS-docs/pull/209 Accessed on: Dec 01, 2023.

[7]

2023. Pull Request #3937. https://github.com/snapshot-labs/snapshot/pull/3937 Accessed on: Dec 01, 2023.

[8]

2023. Pull Request #5602. https://github.com/rancher-sandbox/rancher-desktop/pull/5602 Accessed on: Dec 01, 2023.

[9]

2023. Pull Request #838. https://github.com/CodeIntelligenceTesting/cifuzz/pull/838 Accessed on: Dec 01, 2023.

[10]

2023. Pull Request #897. https://github.com/reworkd/AgentGPT/pull/897 Accessed on: Dec 01, 2023.

[11]

2023. Pull Request#1638. https://github.com/TeamCodeStream/codestream/pull/1638 Accessed on: Dec 01, 2023.

[12]

2023. Pull Request#219. https://github.com/pwncollege/dojo/pull/219 Accessed on: Dec 01, 2023.

[13]

2023. Pull Request#2230. https://github.com/faker-js/faker/pull/2230 Accessed on: Dec 01, 2023.

[14]

2023. Pull Request#2316. https://github.com/validatorjs/validator.js/pull/2316 Accessed on: Dec 01, 2023.

[15]

2023. Pull Request#301. https://github.com/OSRSB/OsrsBot/pull/301 Accessed on: Dec 01, 2023.

[16]

2023. Pull Request#40. https://github.com/rom-py/rompy/pull/40 Accessed on: Dec 01, 2023.

[17]

2023. Pull Request#8902. https://github.com/NixOS/nix/pull/8902 Accessed on: Dec 01, 2023.

[18]

2023. snapshot-labs/snapshot. https://github.com/snapshot-labs/snapshot Accessed on: Dec 01, 2023.

[19]

2023. TeamCodeStream. https://github.com/TeamCodeStream/codestream Accessed on: Dec 01, 2023.

[20]

Aakash Ahmad, Muhammad Waseem, Peng Liang, Mahdi Fahmideh, Mst Shamima Aktar, and Tommi Mikkonen. 2023. Towards human-bot collaborative software architecting with chatgpt. In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering. 279--285.

Digital Library

[21]

Eman Abdullah AlOmar, Moataz Chouchen, Mohamed Wiem Mkaouer, and Ali Ouni. 2022. Code review practices for refactoring changes: An empirical study on openstack. In Proceedings of the 19th International Conference on Mining Software Repositories. 689--701.

Digital Library

[22]

Narjes Bessghaier, Mohammed Sayagh, Ali Ouni, and Mohamed Wiem Mkaouer. 2023. What Constitutes the Deployment and Run-time Configuration System? An Empirical Study on OpenStack Projects. ACM Transactions on Software Engineering and Methodology (2023).

[23]

Virginia Braun and Victoria Clarke. 2012. Thematic analysis. American Psychological Association.

[24]

Tzeng-Ji Chen. 2023. ChatGPT and other artificial intelligence applications speed up scientific writing. Journal of the Chinese Medical Association 86, 4 (2023), 351--353.

[25]

Moataz Chouchen, Ali Ouni, Raula Gaikovina Kula, Dong Wang, Patanamon Thongtanunam, Mohamed Wiem Mkaouer, and Kenichi Matsumoto. 2021. Anti-patterns in modern code review: Symptoms and prevalence. In 2021 IEEE international conference on software analysis, evolution and reengineering (SANER). IEEE, 531--535.

[26]

Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological bulletin 114, 3 (1993), 494.

[27]

Flávia Coelho, Nikolaos Tsantalis, Tiago Massoni, and Everton LG Alves. 2021. An empirical study on refactoring-inducing pull requests. In Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1--12.

Digital Library

[28]

William Jay Conover. 1999. Practical nonparametric statistics. Vol. 350. john wiley & sons.

[29]

Daniela S Cruzes and Tore Dyba. 2011. Recommended steps for thematic synthesis in software engineering. In 2011 international symposium on empirical software engineering and measurement. IEEE, 275--284.

Digital Library

[30]

Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023. Self-collaboration Code Generation via ChatGPT. arXiv preprint arXiv:2304.07590 (2023).

[31]

Hongyang Du, Zonghang Li, Dusit Niyato, Jiawen Kang, Zehui Xiong, Dong In Kim, et al. 2023. Enabling AI-generated content (AIGC) services in wireless edge networks. arXiv preprint arXiv:2301.03220 (2023).

[32]

Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.03533 (2023).

[33]

Abid Haleem, Mohd Javaid, and Ravi Pratap Singh. 2022. An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges. BenchCouncil transactions on benchmarks, standards and evaluations 2, 4 (2022), 100089.

[34]

Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT a good translator? A preliminary study. arXiv preprint arXiv:2301.08745 (2023).

[35]

OpenAI. 2023. Pull Request example:. https://chat.openai.com/share/cb1b4c4e-b2bc-4b1d-9df2-700be0cab72d Accessed on: Dec 01, 2023.

[36]

Partha Pratim Ray. 2023. ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems (2023).

[37]

Jeanine Romano, Jeffrey D Kromrey, Jesse Coraggio, and Jeff Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen'sd for evaluating group differences on the NSSE and other surveys. In annual meeting of the Florida Association of Institutional Research, Vol. 177. 34.

[38]

Danilo Silva, Nikolaos Tsantalis, and Marco Tulio Valente. 2016. Why we refactor? confessions of github contributors. In Proceedings of the 2016 24th acm sigsoft international symposium on foundations of software engineering. 858--870.

Digital Library

[39]

Dong Wang, Tao Xiao, Patanamon Thongtanunam, Raula Gaikovina Kula, and Kenichi Matsumoto. 2021. Understanding shared links and their intentions to meet information needs in modern code review: A case study of the OpenStack and Qt projects. Empirical Software Engineering 26 (2021), 1--32.

Digital Library

[40]

Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023).

[41]

Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. arXiv preprint arXiv:2303.07839 (2023).

[42]

Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. 2023. A brief overview of ChatGPT: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica 10, 5 (2023), 1122--1136.

[43]

Tao Xiao, Christoph Treude, Hideaki Hata, and Kenichi Matsumoto. 2024. DevGPT: Studying Developer-ChatGPT Conversations. In Proceedings of the International Conference on Mining Software Repositories (MSR 2024).

Digital Library

Cited By

Liu BJiang YZhang YNiu NLi GLiu H(2025)Exploring the potential of general purpose LLMs in automated software refactoring: an empirical studyAutomated Software Engineering10.1007/s10515-025-00500-032:1Online publication date: 1-Mar-2025
https://doi.org/10.1007/s10515-025-00500-0

Index Terms

How Do Software Developers Use ChatGPT? An Exploratory Study on GitHub Pull Requests
1. Software and its engineering
  1. Software creation and management
    1. Collaboration in software development

Recommendations

Engaging with AI: An Exploratory Study on Developers' Sharing and Reactions to ChatGPT in GitHub Pull Requests
ASEW '24: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops

ChatGPT, as a representative Foundation Model (FM)-powered tool, has demonstrated significant potential in assisting developers with various software engineering tasks, such as code generation, program repair, and test creation. However, the timing of ...
An Empirical Study on GitHub Pull Requests’ Reactions
The pull request mechanism is commonly used to propose source code modifications and get feedback from the community before merging them into a software repository. On GitHub, practitioners can provide feedback on a pull request by either commenting on ...
Developers assignment for analyzing pull requests
SAC '15: Proceedings of the 30th Annual ACM Symposium on Applied Computing

A new collaboration approach is becoming increasingly common in open-source projects: the pull request model. In this kind of collaboration, developers that do not belong to the core team of a project can submit contributions to the core team. In ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MSR '24: Proceedings of the 21st International Conference on Mining Software Repositories

April 2024

788 pages

ISBN:9798400705878

DOI:10.1145/3643991

Chair:
Diomidis Spinellis,
Program Chair:
Alberto Bacchelli,
Program Co-chair:
Eleni Constantinou

Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 July 2024

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MSR '24

Sponsor:

SIGSOFT

MSR '24: 21st International Conference on Mining Software Repositories

April 15 - 16, 2024

Lisbon, Portugal

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
94
Total Downloads

Downloads (Last 12 months)94
Downloads (Last 6 weeks)31

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu BJiang YZhang YNiu NLi GLiu H(2025)Exploring the potential of general purpose LLMs in automated software refactoring: an empirical studyAutomated Software Engineering10.1007/s10515-025-00500-032:1Online publication date: 1-Mar-2025
https://doi.org/10.1007/s10515-025-00500-0

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten