short-paper

Characteristics of defective infrastructure as code scripts in DevOps

Author:

Akond RahmanAuthors Info & Claims

ICSE '18: Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings

Pages 476 - 479

https://doi.org/10.1145/3183440.3183452

Published: 27 May 2018 Publication History

Abstract

Defects in infrastructure as code (IaC) scripts can have serious consequences for organizations who adopt DevOps. By identifying which characteristics of IaC scripts correlate with defects, we can identify anti-patterns, and help software practitioners make informed decisions on better development and maintenance of IaC scripts, and increase quality of IaC scripts. The goal of this paper is to help practitioners increase the quality of IaC scripts by identifying characteristics of IaC scripts and IaC development process that correlate with defects, and violate security and privacy objectives. We focus on characteristics of IaC scripts and IaC development that (i) correlate with IaC defects, and (ii) violate security and privacy-related objectives namely, confidentiality, availability, and integrity. For our initial studies, we mined open source version control systems from three organizations: Mozilla, Openstack, and Wikimedia, to identify the defect-related characteristics and conduct our case studies. From our empirical analysis, we identify (i) 14 IaC code and four churn characteristics that correlate with defects; and (ii) 12 process characteristics such as, frequency of changes, and ownership of IaC scripts that correlate with defects. We propose the following studies: (i) identify structural characteristics that correlate with defects; (ii) with respect to prediction performance, compare which characteristics of IaC scripts are more correlated with defects; and (iii) identify characteristics that violate security and privacy objectives.

References

[1]

Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald Gall, and Premkumar Devanbu. 2011. Don'T Touch My Code!: Examining the Effects of Ownership on Software Quality. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE '11). ACM, New York, NY, USA, 4--14.

Digital Library

[2]

Kathy Charmaz. 2014. Constructing grounded theory. Sage Publishing, London, UK.

[3]

R. Chillarege, I. S. Bhandari, J. K. Chaar, M. J. Halliday, D. S. Moebus, B. K. Ray, and M. Y. Wong. 1992. Orthogonal defect classification-a concept for in-process measurements. IEEE Transactions on Software Engineering 18, 11 (Nov 1992), 943--956.

Digital Library

[4]

MITRE Corporation. 2017. Common Weakness Enumeration. https://cwe.mitre.org/. (2017). {Online; accessed 10-November-2017}.

[5]

Wei Fu, Tim Menzies, and Xipeng Shen. 2016. Tuning for software analytics: Is it really necessary? Information and Software Technology 76 (2016), 135 -- 146.

Digital Library

[6]

Oliver Hanappi, Waldemar Hummer, and Schahram Dustdar. 2016. Asserting Reliable Convergence for Configuration Management Scripts. SIGPLAN Not. 51, 10 (Oct. 2016), 328--343.

Digital Library

[7]

P. Hudak. 1998. Modular domain specific languages and tools. In Proceedings. Fifth International Conference on Software Reuse (Cat. No.98TB100203). 134--142.

Digital Library

[8]

Jez Humble and David Farley. 2010. Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation (1st ed.). Addison-Wesley Professional.

Digital Library

[9]

Yujuan Jiang and Bram Adams. 2015. Co-evolution of Infrastructure and Source Code: An Empirical Study. In Proceedings of the 12th Working Conference on Mining Software Repositories (MSR '15). IEEE Press, Piscataway, NJ, USA, 45--55.

Digital Library

[10]

Nachiappan Nagappan and Thomas Ball. 2005. Use of Relative Code Churn Measures to Predict System Defect Density. In Proceedings of the 27th International Conference on Software Engineering (ICSE '05). ACM, New York, NY, USA, 284--292.

Digital Library

[11]

Nachiappan Nagappan, Brendan Murphy, and Victor Basili. 2008. The Influence of Organizational Structure on Software Quality: An Empirical Case Study. In Proceedings of the 30th International Conference on Software Engineering (ICSE '08). ACM, New York, NY, USA, 521--530.

Digital Library

[12]

Alberto S. Nunez-Varela, Hector G. Perez-Gonzalez, Francisco E. Martinez-Perez, and Carlos Soubervielle-Montalvo. 2017. Source code metrics: A systematic mapping study. Journal of Systems and Software 128 (2017), 164 -- 197.

Digital Library

[13]

C. Parnin, E. Helms, C. Atlee, H. Boughton, M. Ghattas, A. Glover, J. Holman, J. Micco, B. Murphy, T. Savor, M. Stumm, S. Whitaker, and L. Williams. 2017. The Top 10 Adages in Continuous Deployment. IEEE Software 34, 3 (May 2017), 86--95.

Digital Library

[14]

A. Pecchia and S. Russo. 2012. Detection of Software Failures through Event Logs: An Experimental Study. In 2012 IEEE 23rd International Symposium on Software Reliability Engineering. 31--40.

Digital Library

[15]

Akond Rahman, Asif Partho, David Meder, and Laurie Williams. 2017. Which Factors Influence Practitioners' Usage of Build Automation Tools?. In Proceedings of the 3rd International Workshop on Rapid Continuous Software Engineering (RCoSE '17). IEEE Press, Piscataway, NJ, USA, 20--26.

Digital Library

[16]

Akond Rahman and Laurie Williams. 2018. Characterizing Defective Configuration Scripts Used for Continuous Deployment. In 2018 IEEE International Conference on Software Testing, Verification and Validation (ICST). To appear. Pre-print: http://akondrahman.github.io/papers/icst2018_tm.pdf.

[17]

Rian Shambaugh, Aaron Weiss, and Arjun Guha. 2016. Rehearsal: A Configuration Verification Tool for Puppet. SIGPLAN Not. 51, 6 (June 2016), 416--430.

Digital Library

[18]

Tushar Sharma, Marios Fragkoulis, and Diomidis Spinellis. 2016. Does Your Configuration Code Smell?. In Proceedings of the 13th International Conference on Mining Software Repositories (MSR '16). ACM, New York, NY, USA, 189--200.

Digital Library

[19]

Pang-Ning Tan, Michael Steinbach, and Vipin Kumar. 2005. Introduction to Data Mining, (First Edition). Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

Digital Library

[20]

Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto. 2016. Automated Parameter Optimization of Classification Techniques for Defect Prediction Models. In Proceedings of the 38th International Conference on Software Engineering (ICSE '16). ACM, New York, NY, USA, 321--332.

Digital Library

[21]

John Viega and Gary McGraw. 2011. Building Secure Software: How to Avoid Security Problems the Right Way (Paperback) (Addison-Wesley Professional Computing Series) (1st ed.). Addison-Wesley Professional.

Digital Library

[22]

Feng Zhang, Audris Mockus, Iman Keivanloo, and Ying Zou. 2016. Towards Building a Universal Defect Prediction Model with Rank Transformed Predictors. Empirical Softw. Engg. 21, 5 (Oct. 2016), 2107--2145.

Digital Library

[23]

Thomas Zimmermann, Rahul Premraj, and Andreas Zeller. 2007. Predicting Defects for Eclipse. In Proceedings of the Third International Workshop on Predictor Models in Software Engineering (PROMISE '07). IEEE Computer Society, Washington, DC, USA, 9--15.

Digital Library

Cited By

Karanjai RKasichainula KXu LDiallo NChen LShi W(2024)DIaC: Re-Imagining Decentralized Infrastructure as Code Using BlockchainIEEE Transactions on Network and Service Management10.1109/TNSM.2023.332576821:2(1319-1332)Online publication date: Apr-2024
https://doi.org/10.1109/TNSM.2023.3325768
ÖZDOĞAN ECERAN OÜSTÜNDAĞ M(2023)Systematic Analysis of Infrastructure as Code TechnologiesGazi University Journal of Science Part A: Engineering and Innovation10.54287/gujsa.137330510:4(452-471)Online publication date: 31-Dec-2023
https://doi.org/10.54287/gujsa.1373305
Neharika KLennon R(2022)Investigations into Secure IaC PracticesProceedings of Seventh International Congress on Information and Communication Technology10.1007/978-981-19-1610-6_25(289-303)Online publication date: 27-Jul-2022
https://doi.org/10.1007/978-981-19-1610-6_25
Show More Cited By

Index Terms

Characteristics of defective infrastructure as code scripts in DevOps
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
  2. Software organization and properties
    1. Extra-functional properties
      1. Software reliability

Recommendations

Gang of eight: a defect taxonomy for infrastructure as code scripts
ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Defects in infrastructure as code (IaC) scripts can have serious consequences, for example, creating large-scale system outages. A taxonomy of IaC defects can be useful for understanding the nature of defects, and identifying activities needed to fix and ...
Defect prediction metrics for infrastructure as code scripts in DevOps
ICSE '18: Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings

Use of infrastructure as code (IaC) scripts helps software teams manage their configuration and infrastructure automatically. Information technology (IT) organizations use IaC scripts to create and manage automated deployment pipelines to deliver ...
Security Smells in Ansible and Chef Scripts: A Replication Study
Continuous Special Section: AI and SE

Context: Security smells are recurring coding patterns that are indicative of security weakness and require further inspection. As infrastructure as code (IaC) scripts, such as Ansible and Chef scripts, are used to provision cloud-based servers and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '18: Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings

May 2018

231 pages

ISBN:9781450356633

DOI:10.1145/3183440

Conference Chair:
Michel Chaudron
Chalmers University of Technology, University of Gothenburg, Sweden
,
General Chair:
Ivica Crnkovic
Chalmers University of Technology, University of Gothenburg, Sweden
,
Program Chairs:
Marsha Chechik
University of Toronto, Canada
,
Mark Harman
Facebook and University College London, United Kingdom

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 27 - June 3, 2018

Gothenburg, Sweden

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
452
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)4

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Karanjai RKasichainula KXu LDiallo NChen LShi W(2024)DIaC: Re-Imagining Decentralized Infrastructure as Code Using BlockchainIEEE Transactions on Network and Service Management10.1109/TNSM.2023.332576821:2(1319-1332)Online publication date: Apr-2024
https://doi.org/10.1109/TNSM.2023.3325768
ÖZDOĞAN ECERAN OÜSTÜNDAĞ M(2023)Systematic Analysis of Infrastructure as Code TechnologiesGazi University Journal of Science Part A: Engineering and Innovation10.54287/gujsa.137330510:4(452-471)Online publication date: 31-Dec-2023
https://doi.org/10.54287/gujsa.1373305
Neharika KLennon R(2022)Investigations into Secure IaC PracticesProceedings of Seventh International Congress on Information and Communication Technology10.1007/978-981-19-1610-6_25(289-303)Online publication date: 27-Jul-2022
https://doi.org/10.1007/978-981-19-1610-6_25
Quattrocchi GTamburri D(2022)Predictive maintenance of infrastructure code using “fluid” datasets: An exploratory study on Ansible defect pronenessJournal of Software: Evolution and Process10.1002/smr.248034:11Online publication date: 14-Jun-2022
https://doi.org/10.1002/smr.2480
Zampetti FGeremia SBavota GDi Penta M(2021)CI/CD Pipelines Evolution and Restructuring: A Qualitative and Quantitative Study2021 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME52107.2021.00048(471-482)Online publication date: Sep-2021
https://doi.org/10.1109/ICSME52107.2021.00048
Anandya RRaharjo TSuhanto A(2021)Challenges of DevOps Implementation : A Case Study from Technology Companies in Indonesia2021 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS10.1109/ICIMCIS53775.2021.9699240(108-113)Online publication date: 28-Oct-2021
https://doi.org/10.1109/ICIMCIS53775.2021.9699240
Ganeshan MVigneshwaran P(2020)A Survey on DevOps Techniques Used in Cloud-Based IOT MashupsICT Systems and Sustainability10.1007/978-981-15-8289-9_37(383-393)Online publication date: 15-Dec-2020
https://doi.org/10.1007/978-981-15-8289-9_37
Leite LRocha CKon FMilojicic DMeirelles P(2019)A Survey of DevOps Concepts and ChallengesACM Computing Surveys10.1145/335998152:6(1-35)Online publication date: 14-Nov-2019
https://dl.acm.org/doi/10.1145/3359981
Vassallo CProksch SGall HDi Penta MAtlee JBultan TWhittle J(2019)Automated reporting of anti-patterns and decay in continuous integrationProceedings of the 41st International Conference on Software Engineering10.1109/ICSE.2019.00028(105-115)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.1109/ICSE.2019.00028
Céspedes DAngeleri PMelendez KDávila A(2019)Software Product Quality in DevOps Contexts: A Systematic Literature ReviewTrends and Applications in Software Engineering10.1007/978-3-030-33547-2_5(51-64)Online publication date: 17-Oct-2019
https://doi.org/10.1007/978-3-030-33547-2_5

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten