research-article

What questions do programmers ask about configuration as code?

Authors:

Patrick Morrison,

Laurie WilliamsAuthors Info & Claims

RCoSE '18: Proceedings of the 4th International Workshop on Rapid Continuous Software Engineering

Pages 16 - 22

https://doi.org/10.1145/3194760.3194769

Published: 29 May 2018 Publication History

Abstract

Configuration as code (CaC) tools, such as Ansible and Puppet, help software teams to implement continuous deployment and deploy software changes rapidly. CaC tools are growing in popularity, yet what challenges programmers encounter about CaC tools, have not been characterized. A systematic investigation on what questions are asked by programmers, can help us identify potential technical challenges about CaC, and can aid in successful use of CaC tools. The goal of this paper is to help current and potential configuration as code (CaC) adoptees in identifying the challenges related to CaC through an analysis of questions asked by programmers on a major question and answer website. We extract 2,758 Puppet-related questions asked by programmers from January 2010 to December 2016, posted on Stack Overflow. We apply qualitative analysis to identify the questions programmers ask about Puppet. We also investigate the trends in questions with unsatisfactory answers, and changes in question categories over time. From our empirical study, we synthesize 16 major categories of questions. The three most common question categories are: (i) syntax errors, (ii) provisioning instances; and (iii) assessing Puppet's feasibility to accomplish certain tasks. Three categories of questions that yield the most unsatisfactory answers are (i) installation, (ii) security, and (iii) data separation.

References

[1]

Kartik Bajaj, Karthik Pattabiraman, and Ali Mesbah. 2014. Mining Questions Asked by Web Developers. In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR 2014). ACM, New York, NY, USA, 112--121.

Digital Library

[2]

Anton Barua, Stephen W. Thomas, and Ahmed E. Hassan. 2014. What Are Developers Talking About? An Analysis of Topics and Trends in Stack Overflow. Empirical Softw. Engg. 19, 3 (June 2014), 619--654.

Digital Library

[3]

John L Campbell, Charles Quincy, Jordan Osserman, and Ove K Pedersen. 2013. Coding in-depth semistructured interviews: Problems of unitization and inter-coder reliability and agreement. Sociological Methods & Research 42, 3 (2013), 294--320.

[4]

D. R. Cox and A. Stuart. 1955. Some Quick Sign Tests for Trend in Location and Dispersion. Biometrika 42, 1/2 (1955), 80--95. http://www.jstor.org/stable/2333424

[5]

Stack Exchange. 2017. Stack Exchange. https://data.stackexchange.com/. (2017). {Online; accessed 04-12-2017}.

[6]

Oliver Hanappi, Waldemar Hummer, and Schahram Dustdar. 2016. Asserting Reliable Convergence for Configuration Management Scripts. SIGPLAN Not. 51, 10 (Oct. 2016), 328--343.

Digital Library

[7]

Jez Humble and David Farley. 2010. Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation (1st ed.). Addison-Wesley Professional.

Digital Library

[8]

Yujuan Jiang and Bram Adams. 2015. Co-evolution of Infrastructure and Source Code: An Empirical Study. In Proceedings of the 12th Working Conference on Mining Software Repositories (MSR '15). IEEE Press, Piscataway, NJ, USA, 45--55. http://dl.acm.org/citation.cfm?id=2820518.2820527

Digital Library

[9]

Spencer Krum, William Van Hevelingen, Ben Kero, James Turnbull, and Jeffrey McCune. 2013. Pro Puppet (2nd ed.). Apress, Berkely, CA, USA.

Digital Library

[10]

M. Linares-Vasquez, B. Dit, and D. Poshyvanyk. 2013. An exploratory analysis of mobile development issues using stack overflow. In 2013 10th Working Conference on Mining Software Repositories (MSR). 93--96.

Digital Library

[11]

C. Parnin, E. Helms, C. Atlee, H. Boughton, M. Ghattas, A. Glover, J. Holman, J. Micco, B. Murphy, T. Savor, M. Stumm, S. Whitaker, and L. Williams. 2017. The Top 10 Adages in Continuous Deployment. IEEE Software 34, 3 (May 2017), 86--95.

Digital Library

[12]

Gustavo Pinto, Fernando Castor, and Yu David Liu. 2014. Mining Questions About Software Energy Consumption. In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR 2014). ACM, New York, NY, USA, 22--31.

Digital Library

[13]

Puppet. 2017. Puppet Documentation. https://docs.puppet.com/. (2017). {Online; accessed 04-01-2018}.

[14]

Akond Rahman. 2018. Reference to Examples. http://tiny.cc/example-reffs-so. (2018). {Online; accessed 30-01-2018}.

[15]

Akond Rahman, Asif Partho, David Meder, and Laurie Williams. 2017. Which Factors Influence Practitioners' Usage of Build Automation Tools?. In Proceedings of the 3rd International Workshop on Rapid Continuous Software Engineering (RCoSE '17). IEEE Press, Piscataway, NJ, USA, 20--26.

[16]

Akond Rahman and Laurie Williams. 2018. Characterizing Defective Configuration Scripts Used for Continuous Deployment. In 2018 IEEE International Conference on Software Testing, Verification and Validation (ICST). To appear. Pre-print: http://akondrahman.github.io/papers/icst2018_tm.pdf.

[17]

A. A. U. Rahman, E. Helms, L. Williams, and C. Parnin. 2015. Synthesizing Continuous Deployment Practices Used in Software Development. In 2015 Agile Conference. 1--10.

Digital Library

[18]

M. Reboucas, G. Pinto, F. Ebert, W. Torres, A. Serebrenik, and F. Castor. 2016. An Empirical Study on the Usage of the Swift Programming Language. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. 634--638.

[19]

Christoffer Rosen and Emad Shihab. 2016. What are mobile developers asking about? A large scale study using stack overflow. Empirical Software Engineering 21, 3 (01 Jun 2016), 1192--1223.

Digital Library

[20]

Rian Shambaugh, Aaron Weiss, and Arjun Guha. 2016. Rehearsal: A Configuration Verification Tool for Puppet. SIGPLAN Not. 51, 6 (June 2016), 416--430.

Digital Library

[21]

Tushar Sharma, Marios Fragkoulis, and Diomidis Spinellis. 2016. Does Your Configuration Code Smell?. In Proceedings of the 13th International Conference on Mining Software Repositories (MSR '16). ACM, New York, NY, USA, 189--200.

Digital Library

[22]

Xin-Li Yang, David Lo, Xin Xia, Zhi-Yuan Wan, and Jian-Ling Sun. 2016. What Security Questions Do Developers Ask? A Large-Scale Study of Stack Overflow Posts. Journal of Computer Science and Technology 31, 5 (01 Sep 2016), 910--924.

[23]

T. Zimmermann. 2016. Card-sorting: From text to themes. In Perspectives on Data Science for Software Engineering, Tim Menzies, Laurie Williams, and Thomas Zimmermann (Eds.). Morgan Kaufmann, Boston, 137 -- 141.

Cited By

Nazário MBonifácio RDe Souza CKamei FPinto G(2025)Strategies to Mitigate Configuration Differences in Software Development: A Rapid Review of Grey LiteratureJournal of Software Engineering Research and Development10.5753/jserd.2025.437813:1Online publication date: 20-Feb-2025
https://doi.org/10.5753/jserd.2025.4378
Yasmin JWang JTian YAdams B(2025)An empirical study of developers’ challenges in implementing Workflows as Code: A case study on Apache AirflowJournal of Systems and Software10.1016/j.jss.2024.112248219(112248)Online publication date: Jan-2025
https://doi.org/10.1016/j.jss.2024.112248
Dakić VRedžepagić JBašić MŽgrablić L(2024)Methodology for Automating and Orchestrating Performance Evaluation of Kubernetes Container Network InterfacesComputers10.3390/computers1311028313:11(283)Online publication date: 1-Nov-2024
https://doi.org/10.3390/computers13110283
Show More Cited By

Recommendations

Gang of eight: a defect taxonomy for infrastructure as code scripts
ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Defects in infrastructure as code (IaC) scripts can have serious consequences, for example, creating large-scale system outages. A taxonomy of IaC defects can be useful for understanding the nature of defects, and identifying activities needed to fix and ...
Security Smells in Ansible and Chef Scripts: A Replication Study
Continuous Special Section: AI and SE

Context: Security smells are recurring coding patterns that are indicative of security weakness and require further inspection. As infrastructure as code (IaC) scripts, such as Ansible and Chef scripts, are used to provision cloud-based servers and ...
Does your configuration code smell?
MSR '16: Proceedings of the 13th International Conference on Mining Software Repositories

Infrastructure as Code (IaC) is the practice of specifying computing system configurations through code, and managing them through traditional software engineering methods. The wide adoption of configuration management and increasing size and complexity ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

RCoSE '18: Proceedings of the 4th International Workshop on Rapid Continuous Software Engineering

May 2018

56 pages

ISBN:9781450357456

DOI:10.1145/3194760

Conference Chairs:
Jan Bosch
Chalmers University of Technology | University of Gothenburg, Sweden
,
Brian Fitzgerald
University of Limerick, Ireland
,
Michael Goedicke
University of Duisburg-Essen, Germany
,
Helena Holmström Olsson
Malmö University, Sweden
,
Marco Konersmann
University of Duisburg-Essen, Germany
,
Stephan Krusche
Technical University of Munich, Germany

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 29, 2018

Gothenburg, Sweden

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
507
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)5

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nazário MBonifácio RDe Souza CKamei FPinto G(2025)Strategies to Mitigate Configuration Differences in Software Development: A Rapid Review of Grey LiteratureJournal of Software Engineering Research and Development10.5753/jserd.2025.437813:1Online publication date: 20-Feb-2025
https://doi.org/10.5753/jserd.2025.4378
Yasmin JWang JTian YAdams B(2025)An empirical study of developers’ challenges in implementing Workflows as Code: A case study on Apache AirflowJournal of Systems and Software10.1016/j.jss.2024.112248219(112248)Online publication date: Jan-2025
https://doi.org/10.1016/j.jss.2024.112248
Dakić VRedžepagić JBašić MŽgrablić L(2024)Methodology for Automating and Orchestrating Performance Evaluation of Kubernetes Container Network InterfacesComputers10.3390/computers1311028313:11(283)Online publication date: 1-Nov-2024
https://doi.org/10.3390/computers13110283
Ahmad ASandahl KHasselqvist DSandberg PHong JPark J(2024)Information Needs in Continuous Integration and Delivery in Large Scale Organizations: An Observational StudyProceedings of the 39th ACM/SIGAPP Symposium on Applied Computing10.1145/3605098.3636035(1262-1271)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1145/3605098.3636035
Theodoropoulos TRosa LBenzaid CGray PMarin EMakris ACordeiro LDiego FSorokin PGirolamo MBarone PTaleb TTserpes K(2023)Security in Cloud-Native Services: A SurveyJournal of Cybersecurity and Privacy10.3390/jcp30400343:4(758-793)Online publication date: 26-Oct-2023
https://doi.org/10.3390/jcp3040034
Cogo FXia XHassan A(2023)Assessing the Alignment between the Information Needs of Developers and the Documentation of Programming Languages: A Case Study on RustACM Transactions on Software Engineering and Methodology10.1145/354694532:2(1-48)Online publication date: 4-Apr-2023
https://dl.acm.org/doi/10.1145/3546945
Basak SNeil LReaves BWilliams LGrundy JPollock LPenta M(2023)What Challenges Do Developers Face about Checked-in Secrets in Software Artifacts?Proceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00141(1635-1647)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00141
Begoug MBessghaier NOuni AAlOmar EMkaouer M(2023)What Do Infrastructure-as-Code Practitioners Discuss: An Empirical Study on Stack Overflow2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1109/ESEM56168.2023.10304847(1-12)Online publication date: 26-Oct-2023
https://doi.org/10.1109/ESEM56168.2023.10304847
Tanzil MSarker MUddin GIqbal A(2023)A mixed method study of DevOps challengesInformation and Software Technology10.1016/j.infsof.2023.107244161(107244)Online publication date: Sep-2023
https://doi.org/10.1016/j.infsof.2023.107244
Liu MPeng XMarcus AXing STreude CZhao C(2022)API-Related Developer Information Needs in Stack OverflowIEEE Transactions on Software Engineering10.1109/TSE.2021.312020348:11(4485-4500)Online publication date: 1-Nov-2022
https://doi.org/10.1109/TSE.2021.3120203
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten