skip to main content
10.1145/3666015.3666020acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article
Open access

Fully Automated DORA Metrics Measurement for Continuous Improvement

Published: 04 September 2024 Publication History

Abstract

Many organizations implementing DevOps are adopting DORA metrics to measure their capabilities. Surveys are commonly used to gather the DORA metric data. The data are typically captured only in longer intervals, and on the team level, omitting important details on individual software systems. In this paper, we present a solution that enables completely automated measurement and calculation of the DORA metrics from DevOps tooling on individual microservices level, and in real-time.We evaluated the developed solution in an industrial case study consisting of 37 microservices over a four-week period. The evaluation demonstrated the ability for a completely automated DORA metrics calculation which provides much greater transparency for data-driven decision-making and optimizations on a more fine-granular level allowing teams to continually improve their software processes and outputs.

References

[1]
Marc Andreessen. 2011. Why Software Is Eating The World. Wall Street Journal (2011), 1–5. http://online.wsj.com/article/SB10001424053111903480904576512250915629460.html
[2]
ArgoCD. 203. Argo CD. https://argo-cd.readthedocs.io/en/stable/ https://argo-cd.readthedocs.io/en/stable/ (retrieved: 28.12.2023).
[3]
Atlassian. 2023. Bamboo. https://www.atlassian.com/software/bamboo (retrieved: 28.12.2022).
[4]
Atlassian & CITE. 2020. 2020 DevOps Trends Survey. https://www.atlassian.com/whitepapers/devops-survey-2020
[5]
Robert Biddle, Martin Kropp, Andreas Meier, and Craig Anslow. 2021. Agile Software Development: Practices, Self-Organization, and Satisfaction. The Agile Imperative: Teams, Organizations and Society under Reconstruction? (2021), 39–54.
[6]
Alistair Cockburn and Jim Highsmith. 2001. Agile Software Development: The People Factor. Computer 34, 11 (2001), 131–133.
[7]
Go Community. 2023. Effective Go. https://go.dev/doc/effective_go (retrieved: 27.2.2023).
[8]
Go Community. 2023. golang-standards/project-layout. https://github.com/golang-standards/project-layout (retrieved: 27.2.2023).
[9]
Nicola Dragoni, Saverio Giallorenzo, Alberto Lluch Lafuente, Manuel Mazzara, Fabrizio Montesi, Ruslan Mustafin, and Larisa Safina. 2017. Microservices: Yesterday, Today, and Tomorrow. Present and Ulterior Software Engineering (2017), 195–216. https://doi.org/10.1007/978-3-319-67425-4_12 arxiv:1606.04036
[10]
Christof Ebert, Gorka Gallardo, Josune Hernantes, and Nicolas Serrano. 2016. DevOps. IEEE Software 33, 3 (2016), 94 – 100.
[11]
Kathleen M. Eisenhardt. 1989. Building Theories from Case Study Research. Academy of Management Review 14, 4 (1989), 532–550. https://doi.org/10.5465/amr.1989.4308385
[12]
Norman E. Fenton and Martin Neil. 1999. Software metrics: successes, failures and new directions. Journal of Systems and Software 47, 2 (1999), 149–157. https://doi.org/10.1016/S0164-1212(99)00035-7
[13]
Gustaw Fit. 2022. How Zoopla used DORA Metrics to Boost Deployments, Increase Automation and More. https://about.gitlab.com/blog/2022/01/24/how-zoopla-uses-dora-metrics-and-your-team-can-too/ https://about.gitlab.com/blog/2022/01/24/how-zoopla-uses-dora-metrics-and-your-team-can-too/ (retrieved 12.12.2023).
[14]
Nicole Forsgren, Monica Chiarini Tremblay, Debra Vander Meer, and Jez Humble. 2017. DORA Platform: DevOps Assessment and Benchmarking. International Conference on Design Science Research in Information SystemsMay (2017). https://doi.org/10.1007/978-3-319-59144-5
[15]
Nicole Forsgren, Jez Humble, and Gene Kim. 2018. Accelerate: The Science of Lean Software and DevOps - Building and Scaling High Performing Technology Organizations. IT Revolution.
[16]
Nicole Forsgren and Mik Kersten. 2018. DevOps Metrics. Commun. ACM 61, 4 (2018), 44 – 48. https://doi.org/10.1145/3178368.3182626
[17]
Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler. 2021. The SPACE of Developer Productivity: There’s more to it than you think. Queue 19, 1 (2021), 20–48.
[18]
Nicole Forsgren Velasquez, Gene Kim, Nigel Kersten, and Jez Humble. 2014. 2014 State of DevOps Report. Puppetlabs (2014). http://puppetlabs.com/2014-devops-report
[19]
Yamo Gebrewold and Johanna Wirell. 2023. Challenges with Measuring Software Delivery Performance Metrics: A case study at a software organisation. https://urn.kb.se/resolve?urn=urn:nbn:se:lnu:diva-121933
[20]
Github. 2023. Github Actions. https://docs.github.com/en/actions https://docs.github.com/en/actions (retrieved: 28.12.2023).
[21]
Gitlab. 2023. Gitlab API - Commits. https://docs.gitlab.com/ee/api/commits.html https://docs.gitlab.com/ee/api/commits.html (retrieved: 5.1.2023).
[22]
Gitlab. 2023. Gitlab API - Pipelines. https://docs.gitlab.com/ee/api/pipelines.html https://docs.gitlab.com/ee/api/pipelines.html (retrieved: 26.2.2023).
[23]
Gitlab. 2023. Gitlab CICD. https://docs.gitlab.com/ee/ci/ (retrieved: 28.12.2023).
[24]
Google. 2023. Google Go Style. https://google.github.io/styleguide/go/ (retrieved: 27.2.2023).
[25]
Andrew Harmel-Law. 2022. Four Key Metrics Unleashed. In Software Architecture Metrics : Case Studies to Improve the Quality of Your Architecture, Christian Ciceri, Dave Farley, and Neal Ford (Eds.). O’Reilly, 15 – 42.
[26]
Jim Highsmith and Alistair Cockburn. 2021. Development: The Business of Innovation. Computer 34, 9 (2021), 120–127.
[27]
Jenkins. 2023. Jenkins. https://www.jenkins.io/ https://www.jenkins.io/ (retrieved: 28.12.2023).
[28]
Brendan Julian, James Noble, and Craig Anslow. 2019. Agile Practices in Practice: Towards a Theory of Agile Adoption and Process Evolution. In Agile Processes in Software Engineering and Extreme Programming, Philippe Kruchten, Steven Fraser, and François Coallier (Eds.). Springer International Publishing, Cham, 3–18.
[29]
Marco Kuhrmann, Paolo Tell, Regina Hebig, Jil Klünder, Jürgen Münch, Oliver Linssen, Dietmar Pfahl, Michael Felderer, Christian R. Prause, Stephen G. MacDonell, Joyce Nakatumba-Nabende, David Raffo, Sarah Beecham, Eray Tüzün, Gustavo López, Nicolas Paez, Diego Fontdevila, Sherlock A. Licorish, Steffen Küpper, Günther Ruhe, Eric Knauss, Özden Özcan-Top, Paul Clarke, Fergal McCaffery, Marcela Genero, Aurora Vizcaino, Mario Piattini, Marcos Kalinowski, Tayana Conte, Rafael Prikladnicki, Stephan Krusche, Ahmet Coşkunçay, Ezequiel Scott, Fabio Calefato, Svetlana Pimonova, Rolf-Helge Pfeiffer, Ulrik Pagh Schultz, Rogardt Heldal, Masud Fazal-Baqaie, Craig Anslow, Maleknaz Nayebi, Kurt Schneider, Stefan Sauer, Dietmar Winkler, Stefan Biffl, Maria Cecilia Bastarrica, and Ita Richardson. 2022. What Makes Agile Software Development Agile?IEEE Transactions on Software Engineering 48, 9 (2022), 3523–3539. https://doi.org/10.1109/TSE.2021.3099532
[30]
E. Kupianinen, M. V. Maentylae, and J. Itkonen. 2015. Using Metrics in Agile and Lean Software Development - A Systematic Literature Review of Industrial Studies. Information and Software Technology 62 (2015), 143–163.
[31]
Francesco Lomio, Zadia Codabux, Dale Birtch, Dale Hopkins, and Davide Taibi. 2022. On the Benefits of the Accelerate Metrics: An Industrial Survey at Vendasta. Proceedings - 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) (2022), 46–50. https://doi.org/10.1109/SANER53432.2022.00017
[32]
Aathira Nair. 2022. How the DORA Metrics can help DevOps Team Performance. https://about.gitlab.com/blog/2022/04/20/how-the-dora-metrics-can-help-devops-team-performance/ https://about.gitlab.com/blog/2022/04/20/how-the-dora-metrics-can-help-devops-team-performance/ (retrieved: 12.12.2023).
[33]
Pelorus. 2023. konveyor/pelorus. https://github.com/konveyor/pelorus (retrieved: 26.2.2023).
[34]
Pulasthi Perera, Roshali Silva, and Indika Perera. 2017. Improve Software Quality through Practicing DevOps. Proceedings of the 17th International Conference on Advances in ICT for Emerging Regions (ICTer) (2017). https://doi.org/10.1109/ICTER.2017.8257807
[35]
Claire Peters, Dave Farley, Daniella Villalba, Dave Stanke, Derek DeBellis, Eric Maxwell, John Speed Meyers, Kaiyuan Xu, Nathen Harvey, and Todd Kulesza. 2022. 2022 Accelerate: State of DevOps Report. https://cloud.google.com/devops/state-of-devops
[36]
Prometheus. 2023. Prometheus. https://prometheus.io/ https://prometheus.io/ (retrieved: 16.1.2023).
[37]
Janick Rüegger. 2023. unnmdnwb3/dora. https://github.com/unnmdnwb3/dora (retrieved: 27.2.2023).
[38]
Janick Rüegger. 2023. unnmdnwb3/dora-evaluation. https://github.com/unnmdnwb3/dora-evaluation (retrieved: 27.2.2023).
[39]
Marc Sallin, Martin Kropp, Craig Anslow, James W. Quilty, and Andreas Meier. 2021. Measuring Software Delivery Performance Using the Four Key Metrics of DevOps. XP 2021: Agile Processes in Software Engineering and Extreme Programming 419 (2021), 103 – 122.
[40]
Ken Schwaber and Jeff Sutherland. 2020. The Scrum Guide - The Definitive Guide to Scrum: The Rules of the Game. https://doi.org/10.1002/9781119203278.app2
[41]
Dustin Smith, Daniella Villalba, Michelle Irvine, Dave Stanke, and Nathen Harvey. 2021. Accelerate: State of DevOps 2021. https://services.google.com/fh/files/misc/state-of-devops-2021.pdf?hl=pt-br
[42]
DORA Team. 2023. dora-team/fourkeys. https://github.com/dora-team/fourkeys (retrieved: 26.2.2023).
[43]
Thoughtworks. 2022. Thoughtworks Techradar: March 2022. https://www.thoughtworks.com/radar/techniques/four-key-metrics (retrieved: 24.2.2024).
[44]
Thoughtworks. 2023. thoughtworks/metrik. https://github.com/thoughtworks/metrik (retrieved: 26.2.2023).
[45]
Anna Wiedemann, Nicole Forsgren, Manuel Wiesche, Heiko Gewald, and Helmut Krcmar. 2019. Research for practice: The Devops phenomenon. Commun. ACM 62, 8 (2019), 44–49. https://doi.org/10.1145/3331138
[46]
Liming Zhu, Len Bass, and George Champlin-Scharff. 2016. DevOps and Its Practices. IEEE Software 33, 3 (2016), 32–34. https://doi.org/10.1109/MS.2016.81

Index Terms

  1. Fully Automated DORA Metrics Measurement for Continuous Improvement

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ICSSP '24: Proceedings of the 2024 International Conference on Software and Systems Processes
    September 2024
    106 pages
    ISBN:9798400709913
    DOI:10.1145/3666015
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 04 September 2024

    Check for updates

    Author Tags

    1. Agile methodologies
    2. Continuous improvement
    3. DORA metrics
    4. DevOps practices
    5. Microservices
    6. Software delivery

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    ICSSP '24

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 273
      Total Downloads
    • Downloads (Last 12 months)273
    • Downloads (Last 6 weeks)66
    Reflects downloads up to 20 Jan 2025

    Other Metrics

    Citations

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media