research-article

TerraMetrics: An Open Source Tool for Infrastructure-as-Code (IaC) Quality Metrics in Terraform

Authors:

Moataz Chouchen,

Ali OuniAuthors Info & Claims

ICPC '24: Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension

Pages 450 - 454

https://doi.org/10.1145/3643916.3644439

Published: 13 June 2024 Publication History

Abstract

Infrastructure-as-Code (IaC) constitutes a pivotal DevOps methodology, leading edge of software deployment onto cloud platforms. IaC relies on source code files rather than manual configuration to manage the infrastructure of a software system. Terraform, an IaC tool and its declarative configuration language named HCL, has recently garnered considerable attention among IaC practitioners. Like other software artefacts, Terraform files could be affected by misconfigurations, faults, and smells. Therefore, DevOps practitioners might benefit from a quality assurance tool to help them perform quality assurance activities on Terrafrom artefacts. This paper introduces TerraMetrics, an open-source tool designed to characterize the quality of Terraform artefacts by providing a catalogue of 40 quality metrics. TerraMetrics leverages the Terraform Abstract Syntax Tree (AST) to extract the metric list, offering a potentially enduring solution compared to conventional regular expressions. This tool comprises three main components: (i) a parser transforming HCL code into an AST, (ii) visitors that traverse the AST nodes to extract the metrics, and (iii) collectors for storing the collected metrics in JSON format. The TerraMetrics tool is publicly available as an Open Source tool, with a demo video, at: https://github.com/stilab-ets/terametrics.

References

[1]

[n. d.]. Amazon Web Services. https://aws.amazon.com/. Accessed on: Nov 1, 2023.

[2]

[n. d.]. Azure. https://azure.microsoft.com/en-ca. Accessed on: Nov 1, 2023.

[3]

[n. d.]. Cloud Google Platform. https://cloud.google.com/. Accessed on: Nov 1, 2023.

[4]

[n. d.]. Find Bugs in Java Programs. https://findbugs.sourceforge.net/. Accessed on: Nov 1, 2023.

[5]

[n. d.]. PMD Source Code Analyzer. https://pmd.github.io/. Accessed on: Nov 1, 2023.

[6]

[n. d.]. Terraform Language Documentation. https://developer.hashicorp.com/terraform/language. Accessed on: Nov 1, 2023.

[7]

[n. d.]. TerraMetrics: An Open Source Tool for Infrastructure-as-Code (IaC) Quality Metrics in Terraform. https://github.com/stilab-ets/terametrics. Accessed on: Nov 1, 2023.

[8]

[n. d.]. The top programming languages. https://octoverse.github.com/2022/top-programming-languages. Accessed on: Nov 1, 2023.

[9]

aquasecurity. [n. d.]. https://github.com/aquasecurity/tfsec. Accessed on: Nov 1, 2023.

[10]

Martin Atkins. [n. d.]. HCL Syntax-Agnostic Information Model. https://github.com/hashicorp/hcl/blob/main/spec.md. Accessed on: Nov 1, 2023.

[11]

Beck. 2002. Test Driven Development: By Example. Addison-Wesley Longman Publishing Co., Inc., USA.

[12]

Mahi Begoug, Narjes Bessghaier, Ali Ouni, Eman Alomar, and Mohamed Wiem Mkaouer. 2023. What Do Infrastructure-as-Code Practitioners Discuss: An Empirical Study on Stack Overflow. In 17th International Conference on Empirical Software Engineering and Measurement (ESEM). 1--12 pages.

[13]

Mahi Begoug, Moataz Chouchen, Ali Ouni, Eman Alomar, and Mohamed Wiem Mkaouer. 2024. Fine-Grained Just-In-Time Defect Prediction at the Block Level in Infrastructure-as-Code (IaC). In ACM International Conference on Mining Software Repositories (MSR). 1--12 pages.

[14]

Nemania Borovits, Indika Kumara, Parvathy Krishnan, Stefano Dalla Palma, Dario Di Nucci, Fabio Palomba, Damian A Tamburri, and Willem-Jan van den Heuvel. 2020. DeepIaC: deep learning-based linguistic anti-pattern detection in IaC. In Proceedings of the 4th ACM SIGSOFT International Workshop on Machine-Learning Techniques for Software-Quality Evaluation. 7--12.

[15]

Yevgeniy Brikman. 2022. Terraform: Up and Running. " O'Reilly Media, Inc.".

[16]

Checkov. [n. d.]. https://github.com/bridgecrewio/checkov. Accessed on: Nov 1, 2023.

[17]

Stefano Dalla Palma, Dario Di Nucci, Fabio Palomba, and Damian Andrew Tamburri. 2020. Toward a catalog of software quality metrics for infrastructure code. Journal of Systems and Software 170 (2020), 110726.

[18]

Stefano Dalla Palma, Dario Di Nucci, and Damian A Tamburri. 2020. Ansible-Metrics: A Python library for measuring Infrastructure-as-Code blueprints in Ansible. SoftwareX 12 (2020), 100633.

[19]

GitHub. [n. d.]. The State of Open Source and AI. https://github.blog/2023-11-08-the-state-of-open-source-and-ai/. Accessed on: Nov 12, 2023.

[20]

hashicorp. [n. d.]. https://www.terraform.io/. Accessed on: Nov 1, 2023.

[21]

Red Hat. [n. d.]. https://www.ansible.com/. Accessed on: Nov 1, 2023.

[22]

Richard Helm, Ralph E Johnson, Erich Gamma, and John Vlissides. 2000. Design patterns: Elements of reusable object-oriented software. Braille Jymico Incorporated Quebec.

[23]

Pandu Ranga Reddy Konala, Vimal Kumar, and David Bainbridge. 2023. SoK: Static Configuration Analysis in Infrastructure as Code Scripts. In 2023 IEEE International Conference on Cyber Security and Resilience (CSR). IEEE, 281--288.

[24]

Luigi Lavazza, Sandro Morasca, and Marco Gatto. 2023. An empirical study on software understandability and its dependence on code characteristics. Empirical Software Engineering 28, 6 (2023), 1--24.

Digital Library

[25]

Valentina Lenarduzzi, Terhi Kilamo, and Andrea Janes. 2023. Does Cyclomatic or Cognitive Complexity Better Represents Code Understandability? An Empirical Investigation on the Developers Perception. arXiv preprint arXiv:2303.07722 (2023).

[26]

Alisdair McDiarmid. [n. d.]. HCL Native Syntax Specification. https://github.com/hashicorp/hcl/blob/main/hclsyntax/spec.md. Accessed on: Nov 1, 2023.

[27]

Kief Morris and Brice Thompson. 2020. Infrastructure as Code (2nd ed.). O'Reilly Media.

[28]

Ruben Opdebeeck, Ahmed Zerouali, and Coen De Roover. 2023. Control and data flow in security smell detection for infrastructure as code: Is it worth the effort?. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). IEEE, 534--545.

[29]

Sneh Pandya and Riya Guha Thakurta. 2022. Hands-on Infrastructure as Code with Hashicorp Terraform. In Introduction to Infrastructure as Code: A Brief Guide to the Future of DevOps. Springer, 99--133.

[30]

Perforce. [n. d.]. https://www.puppet.com/. Accessed on: Nov 1, 2023.

[31]

Anthony Peruma, Khalid Almalki, Christian D Newman, Mohamed Wiem Mkaouer, Ali Ouni, and Fabio Palomba. 2020. Tsdetect: An open source test smells detection tool. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 1650--1654.

Digital Library

[32]

Akond Rahman, Md Rayhanur Rahman, Chris Parnin, and Laurie Williams. 2021. Security smells in ansible and chef scripts: A replication study. ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 1 (2021), 1--31.

Digital Library

[33]

runterrascan. [n. d.]. https://github.com/tenable/terrascan. Accessed on: Nov 1, 2023.

[34]

Nuno Saavedra and João F Ferreira. 2022. GLITCH: an Intermediate-Representation-Based Security Analysis for Infrastructure as Code Scripts. arXiv preprint arXiv:2205.14371 (2022).

[35]

scitools. [n. d.]. Understand Software Metrics. https://documentation.scitools.com/pdf/metricsdoc.pdf. Accessed on: Nov 1, 2023.

[36]

Tushar Sharma. [n. d.]. Smell detection tool for Puppet code. https://github.com/tushartushar/Puppeteer. Accessed on: Nov 1, 2023.

[37]

Tushar Sharma, Marios Fragkoulis, and Diomidis Spinellis. 2016. Does your configuration code smell?. In Proceedings of the 13th International Conference on Mining Software Repositories. 189--200.

Digital Library

[38]

Dave Smith. [n. d.]. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15999. Accessed on: Nov 1, 2023.

[39]

SonarSource. [n. d.]. https://github.com/SonarSource/sonar-iac/tree/master/iac-extensions/terraform. Accessed on: Nov 1, 2023.

[40]

Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. Fam med 37, 5 (2005), 360--363.

Cited By

Verdet AHamdaqa MSilva LKhomh F(2025)Assessing the adoption of security policies by developers in terraform across different cloud providersEmpirical Software Engineering10.1007/s10664-024-10610-030:3Online publication date: 27-Feb-2025
https://doi.org/10.1007/s10664-024-10610-0
Begoug MOuni A(2024)How Do Infrastructure-as-Code Practitioners Update their Provider Dependencies? An Empirical Study on the AWS ProviderService-Oriented Computing10.1007/978-981-96-0808-9_28(373-388)Online publication date: 7-Dec-2024
https://doi.org/10.1007/978-981-96-0808-9_28

Recommendations

Fine-Grained Just-In-Time Defect Prediction at the Block Level in Infrastructure-as-Code (IaC)
MSR '24: Proceedings of the 21st International Conference on Mining Software Repositories

Infrastructure-as-Code (IaC) is an emerging software engineering practice that leverages source code to facilitate automated configuration of software systems' infrastructure. IaC files are typically complex, containing hundreds of lines of code and ...
Polyglot Code Smell Detection for Infrastructure as Code with GLITCH
ASE '23: Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering

This paper presents GLITCH, a new technology-agnostic framework that enables automated polyglot code smell detection for Infrastructure as Code scripts. GLITCH uses an intermediate representation on which different code smell detectors can be defined. It ...
Source code metrics

Three major programming paradigms measured by source code metrics were identified.The CK metrics and the object oriented paradigm are the most studied subjects.Java benchmark systems are the most commonly measured systems in research.Technology on ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPC '24: Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension

April 2024

487 pages

ISBN:9798400705861

DOI:10.1145/3643916

Chair:
Igor Steinmacher,
Co-chair:
Mario Linares-Vasquez,
Program Chair:
Kevin Patrick Moran,
Program Co-chair:
Olga Baysal

Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2024

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Natural Sciences and Engineering Research Council of Canada

Conference

ICPC '24

Sponsor:

SIGSOFT

ICPC '24: 32nd IEEE/ACM International Conference on Program Comprehension

April 15 - 16, 2024

Lisbon, Portugal

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
181
Total Downloads

Downloads (Last 12 months)181
Downloads (Last 6 weeks)19

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Verdet AHamdaqa MSilva LKhomh F(2025)Assessing the adoption of security policies by developers in terraform across different cloud providersEmpirical Software Engineering10.1007/s10664-024-10610-030:3Online publication date: 27-Feb-2025
https://doi.org/10.1007/s10664-024-10610-0
Begoug MOuni A(2024)How Do Infrastructure-as-Code Practitioners Update their Provider Dependencies? An Empirical Study on the AWS ProviderService-Oriented Computing10.1007/978-981-96-0808-9_28(373-388)Online publication date: 7-Dec-2024
https://doi.org/10.1007/978-981-96-0808-9_28

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten