skip to main content
10.1145/3660829.3660835acmconferencesArticle/Chapter ViewAbstractPublication PagesprogrammingConference Proceedingsconference-collections
research-article
Open access

Ironies of Programming Automation: Exploring the Experience of Code Synthesis via Large Language Models

Published: 09 July 2024 Publication History

Abstract

The widespread availability of large language models (LLMs) has presented the opportunity for novice programmers to make use of them for the purpose of understanding and synthesising code. In this paper, we discuss a small pilot study intended to explore the user experience of doing so in a limited way, and the attitudes of a group of novice programmers towards this style of programming. We also draw parallels to the seminal work of Lisanne Bainbridge, and discuss the way in which her "ironies of automation" are also present when attempting to automate the activity of programming.

References

[1]
Lisanne Bainbridge. 1983. Ironies of automation. In Analysis, design and evaluation of man–machine systems. Elsevier, 129–135.
[2]
Alessio Bucaioni, Hampus Ekedahl, Vilma Helander, and Phuong T. Nguyen. 2024. Programming with ChatGPT: How far can we go?Machine Learning with Applications 15 (2024), 100526. https://doi.org/10.1016/j.mlwa.2024.100526
[3]
Luke Church, Emma Söderberg, and Alan T. McCabe. 2021. Breaking down and making up - a lens for conversing with compilers. In Proceedings of the 32nd Annual Workshop of the Psychology of Programming Interest Group.
[4]
Haytham Hijazi, Ricardo Couceiro, João Castelhano, Paulo De Carvalho, Miguel Castelo-Branco, and Henrique Madeira. 2021. Intelligent Biofeedback Augmented Content Comprehension (TellBack). IEEE Access 9 (2021), 28393–28406. https://doi.org/10.1109/ACCESS.2021.3058664
[5]
Jenny T Liang, Chenyang Yang, and Brad A Myers. 2024. A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges. In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE.
[6]
Chao Liu, Xuanlin Bao, Hongyu Zhang, Neng Zhang, Haibo Hu, Xiaohong Zhang, and Meng Yan. 2023. Improving ChatGPT Prompt for Code Generation. arxiv:2305.08360 [cs.SE]
[7]
Norman H Mackworth 1950. Researches on the measurement of human performance.Researches on the Measurement of Human Performance.268 (1950).
[8]
Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2022. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. arxiv:2211.02265 [cs.SE]
[9]
Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence?arxiv:2208.06213 [cs.HC]
[10]
Richard Sennett. 2008. The craftsman. Yale University Press.
[11]
Janet Siegmund, Christian Kästner, Jörg Liebig, Sven Apel, and Stefan Hanenberg. 2014. Measuring and Modeling Programming Experience. Empirical Softw. Engg. 19, 5 (oct 2014), 1299–1334. https://doi.org/10.1007/s10664-013-9286-4
[12]
Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F. Bissyandé. 2023. Is ChatGPT the Ultimate Programming Assistant – How far is it?arxiv:2304.11938 [cs.SE]
[13]
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arxiv:2302.11382 [cs.SE]
[14]
Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design. arxiv:2303.07839 [cs.SE]
[15]
J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages. https://doi.org/10.1145/3544548.3581388

Cited By

View all
  • (2024)LLM4VV: Exploring LLM-as-a-Judge for Validation and Verification TestsuitesProceedings of the SC '24 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis10.1109/SCW63240.2024.00238(1885-1893)Online publication date: 17-Nov-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
Programming '24: Companion Proceedings of the 8th International Conference on the Art, Science, and Engineering of Programming
March 2024
159 pages
ISBN:9798400706349
DOI:10.1145/3660829
This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 July 2024

Check for updates

Author Tags

  1. code comprehension
  2. human-computer interaction
  3. large language models
  4. prompt engineering
  5. prompt programming

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • Swedish Foundation for Strategic Research
  • Swedish Research Council

Conference

‹Programming› '24
Sponsor:

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)176
  • Downloads (Last 6 weeks)33
Reflects downloads up to 22 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)LLM4VV: Exploring LLM-as-a-Judge for Validation and Verification TestsuitesProceedings of the SC '24 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis10.1109/SCW63240.2024.00238(1885-1893)Online publication date: 17-Nov-2024

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media