A tool supporting root cause analysis for synchronous retrospectives in distributed software teams

https://doi.org/10.1016/j.infsof.2014.01.004Get rights and content

Abstract

Context

Root cause analysis (RCA) is a useful practice for software project retrospectives, and is typically carried out in synchronous collocated face-to-face meetings. Conducting RCA with distributed teams is challenging, as face-to-face meetings are infeasible. Lack of adequate real-time tool support exacerbates this problem. Furthermore, there are no empirical studies on using RCA in synchronous retrospectives of geographically distributed teams.

Objective

This paper presents a real-time cloud-based software tool (ARCA-tool) we developed to support RCA in distributed teams and its initial empirical evaluation. The feasibility of using RCA with distributed teams is also evaluated.

Method

We compared our tool with 35 existing RCA software tools. We conducted field studies of four distributed agile software teams at two international software product companies. The teams conducted RCA collaboratively in synchronous retrospective meetings by using the tool we developed. We collected the data using observations, interviews and questionnaires.

Results

Comparison revealed that none of the existing 35 tools matched all the features of our ARCA-tool. The team members found ARCA-tool to be an essential part of their distributed retrospectives. They considered the software as efficient and very easy to learn and use. Additionally, the team members perceived RCA to be a vital part of the retrospectives. In contrast to the prior retrospective practices of the teams, the introduced RCA method was evaluated as efficient and easy to use.

Conclusion

RCA is a useful practice in synchronous distributed retrospectives. However, it requires software tool support for enabling real-time view and co-creation of a cause-effect diagram. ARCA-tool supports synchronous RCA, and includes support for logging problems and causes, problem prioritization, cause-effect diagramming, and logging of process improvement proposals. It enables conducting RCA in distributed retrospectives.

Introduction

Retrospectives, also known as post-mortems, are activities where the team members share experiences about problems and their causes [1], analyzing a recently ended project and/or iteration. Root cause analysis (RCA) is a structured investigation of a problem to detect which underlying causes need to be solved [2], and a useful practice for retrospectives [3], [4], [5]. Retrospectives are typically conducted in face-to-face meetings, in which the team members first identify problems that occurred. Subsequently, they conduct lightweight RCA by collaboratively creating a cause-effect diagram visualizing the causes of problems [5].

Global software engineering, employing geographically distributed teams, has become a standard way of operating in today’s business [6]. This way of working creates new challenges related to geographical, temporal, cultural and organizational distance [7]. The use of distributed teams also creates a major challenge for conducting team retrospectives [8]. In previous work, we developed a lightweight focus group based RCA method, ARCA, and evaluated it in four industrial field studies using collocated teams [9]. Even though the method was well liked, the companies pointed out the need to conduct RCA with their distributed teams. Literature on distributed retrospectives identifies a similar need and discusses the use of a combination of email, spreadsheets and an online audio bridge to help facilitate the retrospectives [8]. However, relying on such tools in focus group based synchronous RCA is not feasible, as organizing and interpreting a high number of causes using emails and spreadsheets would be highly difficult. Instead, cause-effect diagrams [9] supporting real-time online environment should be used in distributed retrospectives.

There are many proprietary software tools for RCA.1 However, we have not succeeded in finding a web-based tool that fulfills the needs of conducting lightweight RCA in synchronous distributed software project retrospectives. First, the tool should make it possible for RCA participants to co-create a cause-effect diagram [5], [9], which stays in-sync between the sites. Second, the tool should allow the development of process improvement ideas for the causes and maintain links between the improvement ideas and the detected causes [10], [11], [12], [13], [14]. Third, the tool should make it possible to vote on the most severe causes and best improvement ideas [9]. Fourth, the tool should also make it possible to capture and refine the findings of several retrospectives, in order to support organizational learning and knowledge management [3]. To the authors’ best knowledge2 the most frequently lacking feature of current software tools for RCA is the syncing mechanisms needed for simultaneous co-creation of cause-effect diagrams, see Table 1. There are tools for simultaneous graph drawing, e.g., Google Docs drawings [15], but these tools lack features to support RCA, e.g. automatically capturing and refining the findings of retrospectives.

Furthermore, to our knowledge, there are no empirical studies on the feasibility of using RCA in synchronous distributed retrospectives. While there is ample evidence for the benefits of RCA to detect the causes of problems and make improvements in various contexts [9], [10], [11], [12], [13], [16], [17], [18], [19], [20], [21], the existing studies have been conducted in a face-to-face context. Thus, in order to contribute to the existing studies, we developed an online tool for supporting synchronous RCA in distributed software project retrospectives called ARCA-tool.3 It provides features for distributed RCA, idea development, and capturing the lessons learned in many retrospectives.

The goals of this paper are to present ARCA-tool including its technology and main features, and to provide an empirical evaluation of the tool and synchronous RCA in the context of industrial software development with agile teams. In order to evaluate the usefulness of RCA and ARCA-tool, we used interviews, questionnaires, and observations in the retrospectives of geographically distributed industrial software teams, that followed the Scrum methodology [22]. Our research questions were:

  • RQ1: Is ARCA-tool perceived as useful in the distributed retrospectives of agile software teams?

  • RQ2: Is ARCA-tool perceived as easy to use in the distributed retrospectives of agile software teams?

  • RQ3: Is RCA perceived as a good approach to use in the distributed retrospectives of agile software teams?

While the first two questions are related directly to ARCA-tool, we evaluate the RCA method, since the evaluators might have difficulty separating the effect of the tool and the context in which it was applied, i.e. the synchronous retrospective method used and the company context. Naturally, ARCA-tool can be used without the retrospective with the RCA method and vice versa.

The rest of the paper is structured in the following way. Section 2 covers the related work and identifies a gap in research, which is then filled by introducing ARCA-tool in Section 3. Section 4 explains the field study method used to evaluate the tool in real industrial contexts and the results of this evaluation are given in Section 5. Finally, Section 6 contains the discussion and Section 7 provides conclusions and directions for further work.

Section snippets

Related work

In this section, we introduce the concept of software project retrospectives and present problems related to conducting RCA with distributed software teams. We also compare RCA software tools that we have found.

ARCA-tool

This section provides an overview of ARCA-tool. We will discuss how the tool supports distributed retrospectives and the features it includes.

Field study methodology

For the empirical evaluation of ARCA-tool, we used a field study method [37] that allowed us to study the adoption and use of the tool in a real industrial setting. We observed and video recorded four retrospectives conducted by four teams in two companies. After the retrospectives, all participants completed a questionnaire, and selected case participants were interviewed. Thus, we present a rich data set from four industrial software teams, but in contrast to a controlled experiment, we

Results

In this section, we present the field study results. Feedback from ARCA-tool (see Section 3) is presented in Section 5.1 and the feedback from the retrospective method including the RCA method (see Section 4.2) is summarized in Section 5.2. Furthermore, Table 3 summarizes the feedback from the questionnaires, and Table 4, Table 5 summarize the results from the interviews. The tables separate the results regarding the research questions. While RQ1 and RQ2 aim to evaluate ARCA- tool, RQ3

Discussion

In this section, we answer the research questions and discuss our findings and possible threats to the validity of this study.

Conclusions and future work

This paper proposed a real-time cloud-based tool for solving the problem of being infeasible to conduct collocated retrospectives in geographically distributed software teams [8]. ARCA-tool enables conducting collocated and distributed retrospectives with RCA. The most important feature of the tool is the up-to-date real-time view of the retrospective outcome. Additionally, the tool provides features for the co-creation of cause-effect diagrams, the development of improvement ideas, the voting

Acknowledgements

The authors would like to thank the companies participating in the field studies and software engineering students implementing ARCA-tool, in alphabetical order: Helin Anssi Matti, Hovi Roope, Jaanto Jari, Kekäle Mika, Kere Markus, Koistinen Joona, Laukkanen Eero, Patana Jussi, Rihtniemi Pekka, Saarinen Jerome, Sevenius Toni, Valjus Mikko, and Viitanen Jonne.

References (46)

  • T. Jaanu, M. Paasivaara, C. Lassenius, Near-synchronity and distance: instant messaging as a medium for global software...
  • J. Terzakis

    Virtual retrospectives for geographically dispersed software teams

    IEEE Softw.

    (2011)
  • D.N. Card

    Learning from our mistakes with defect causal analysis

    IEEE Softw.

    (1998)
  • M. Leszak, D.E. Perry, D. Stoll, A case study in root cause defect analysis, in: Proceedings of the 2000 International...
  • R.B. Grady

    Software failure analysis for high-return process improvement decisions

    Hewlett-Packard J.

    (1996)
  • P. Jalote, N. Agrawal, Using defect analysis feedback for improving quality and productivity in iterative software...
  • B. Andersen, T. Fagerhaug (Eds.), Root Cause Analysis: Simplified Tools and Techniques, Tony A. William American...
  • News and notes from the Google Drive and Docs teams, “Introducing Google Docs drawings”, Docs Blog, 13th April...
  • M. Kalinowski, G.H. Travassos, D.N. Card, Towards a defect prevention based process improvement approach, in:...
  • R.G. Mays

    Applications of defect prevention in software development

    IEEE J. Sel. Areas Commun.

    (1990)
  • T. Stålhane, Root Cause Analysis and Gap Analysis – A Tale of Two Methods, EuroSPI 2004, Trondheim, Norway, 2004, pp....
  • I. Bhandari et al.

    A case study of software process improvement during development

    IEEE Trans. Softw. Eng.

    (1993)
  • Z.X. Jin, J. Hajdukiewicz, G. Ho, D. Chan, Y. Kow, Using root cause data analysis for requirements and knowledge...
  • Cited by (9)

    • Diagrams or structural lists in software project retrospectives - An experimental comparison

      2015, Journal of Systems and Software
      Citation Excerpt :

      Similarly, even though the use of CED has been introduced as an important part of retrospectives (Bjørnson et al., 2009), the company practices seem to favor textual notations to visualize the retrospective findings (Lehtinen et al., 2011, 2014b). Software tool support for collaborative cause–effect diagramming is also widely missing (Lehtinen et al., 2014b) and therefore using CEDs in the distributed settings is practically challenging. Thus, in terms of the tool support for modern distributed software project retrospectives, we should also determine how to visualize the outcome of RCA.

    • Categorization of risk factors for distributed agile projects

      2015, Information and Software Technology
    • Competencies for Managing Activities in Agile Projects

      2022, Global Journal of Flexible Systems Management
    • Technical and Managerial Difficulties in Postmortem Analysis in Software Projects

      2019, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    View all citing articles on Scopus
    View full text