A tool supporting root cause analysis for synchronous retrospectives in distributed software teams
Introduction
Retrospectives, also known as post-mortems, are activities where the team members share experiences about problems and their causes [1], analyzing a recently ended project and/or iteration. Root cause analysis (RCA) is a structured investigation of a problem to detect which underlying causes need to be solved [2], and a useful practice for retrospectives [3], [4], [5]. Retrospectives are typically conducted in face-to-face meetings, in which the team members first identify problems that occurred. Subsequently, they conduct lightweight RCA by collaboratively creating a cause-effect diagram visualizing the causes of problems [5].
Global software engineering, employing geographically distributed teams, has become a standard way of operating in today’s business [6]. This way of working creates new challenges related to geographical, temporal, cultural and organizational distance [7]. The use of distributed teams also creates a major challenge for conducting team retrospectives [8]. In previous work, we developed a lightweight focus group based RCA method, ARCA, and evaluated it in four industrial field studies using collocated teams [9]. Even though the method was well liked, the companies pointed out the need to conduct RCA with their distributed teams. Literature on distributed retrospectives identifies a similar need and discusses the use of a combination of email, spreadsheets and an online audio bridge to help facilitate the retrospectives [8]. However, relying on such tools in focus group based synchronous RCA is not feasible, as organizing and interpreting a high number of causes using emails and spreadsheets would be highly difficult. Instead, cause-effect diagrams [9] supporting real-time online environment should be used in distributed retrospectives.
There are many proprietary software tools for RCA.1 However, we have not succeeded in finding a web-based tool that fulfills the needs of conducting lightweight RCA in synchronous distributed software project retrospectives. First, the tool should make it possible for RCA participants to co-create a cause-effect diagram [5], [9], which stays in-sync between the sites. Second, the tool should allow the development of process improvement ideas for the causes and maintain links between the improvement ideas and the detected causes [10], [11], [12], [13], [14]. Third, the tool should make it possible to vote on the most severe causes and best improvement ideas [9]. Fourth, the tool should also make it possible to capture and refine the findings of several retrospectives, in order to support organizational learning and knowledge management [3]. To the authors’ best knowledge2 the most frequently lacking feature of current software tools for RCA is the syncing mechanisms needed for simultaneous co-creation of cause-effect diagrams, see Table 1. There are tools for simultaneous graph drawing, e.g., Google Docs drawings [15], but these tools lack features to support RCA, e.g. automatically capturing and refining the findings of retrospectives.
Furthermore, to our knowledge, there are no empirical studies on the feasibility of using RCA in synchronous distributed retrospectives. While there is ample evidence for the benefits of RCA to detect the causes of problems and make improvements in various contexts [9], [10], [11], [12], [13], [16], [17], [18], [19], [20], [21], the existing studies have been conducted in a face-to-face context. Thus, in order to contribute to the existing studies, we developed an online tool for supporting synchronous RCA in distributed software project retrospectives called ARCA-tool.3 It provides features for distributed RCA, idea development, and capturing the lessons learned in many retrospectives.
The goals of this paper are to present ARCA-tool including its technology and main features, and to provide an empirical evaluation of the tool and synchronous RCA in the context of industrial software development with agile teams. In order to evaluate the usefulness of RCA and ARCA-tool, we used interviews, questionnaires, and observations in the retrospectives of geographically distributed industrial software teams, that followed the Scrum methodology [22]. Our research questions were:
RQ1: Is ARCA-tool perceived as useful in the distributed retrospectives of agile software teams?
RQ2: Is ARCA-tool perceived as easy to use in the distributed retrospectives of agile software teams?
RQ3: Is RCA perceived as a good approach to use in the distributed retrospectives of agile software teams?
While the first two questions are related directly to ARCA-tool, we evaluate the RCA method, since the evaluators might have difficulty separating the effect of the tool and the context in which it was applied, i.e. the synchronous retrospective method used and the company context. Naturally, ARCA-tool can be used without the retrospective with the RCA method and vice versa.
The rest of the paper is structured in the following way. Section 2 covers the related work and identifies a gap in research, which is then filled by introducing ARCA-tool in Section 3. Section 4 explains the field study method used to evaluate the tool in real industrial contexts and the results of this evaluation are given in Section 5. Finally, Section 6 contains the discussion and Section 7 provides conclusions and directions for further work.
Section snippets
Related work
In this section, we introduce the concept of software project retrospectives and present problems related to conducting RCA with distributed software teams. We also compare RCA software tools that we have found.
ARCA-tool
This section provides an overview of ARCA-tool. We will discuss how the tool supports distributed retrospectives and the features it includes.
Field study methodology
For the empirical evaluation of ARCA-tool, we used a field study method [37] that allowed us to study the adoption and use of the tool in a real industrial setting. We observed and video recorded four retrospectives conducted by four teams in two companies. After the retrospectives, all participants completed a questionnaire, and selected case participants were interviewed. Thus, we present a rich data set from four industrial software teams, but in contrast to a controlled experiment, we
Results
In this section, we present the field study results. Feedback from ARCA-tool (see Section 3) is presented in Section 5.1 and the feedback from the retrospective method including the RCA method (see Section 4.2) is summarized in Section 5.2. Furthermore, Table 3 summarizes the feedback from the questionnaires, and Table 4, Table 5 summarize the results from the interviews. The tables separate the results regarding the research questions. While RQ1 and RQ2 aim to evaluate ARCA- tool, RQ3
Discussion
In this section, we answer the research questions and discuss our findings and possible threats to the validity of this study.
Conclusions and future work
This paper proposed a real-time cloud-based tool for solving the problem of being infeasible to conduct collocated retrospectives in geographically distributed software teams [8]. ARCA-tool enables conducting collocated and distributed retrospectives with RCA. The most important feature of the tool is the up-to-date real-time view of the retrospective outcome. Additionally, the tool provides features for the co-creation of cause-effect diagrams, the development of improvement ideas, the voting
Acknowledgements
The authors would like to thank the companies participating in the field studies and software engineering students implementing ARCA-tool, in alphabetical order: Helin Anssi Matti, Hovi Roope, Jaanto Jari, Kekäle Mika, Kere Markus, Koistinen Joona, Laukkanen Eero, Patana Jussi, Rihtniemi Pekka, Saarinen Jerome, Sevenius Toni, Valjus Mikko, and Viitanen Jonne.
References (46)
Postmortem reviews: purpose and approaches in software engineering
Inf. Softw. Technol.
(2005)- et al.
Improving the effectiveness of root cause analysis in post mortem analysis: a controlled experiment
Inf. Softw. Technol.
(2009) - et al.
Development and evaluation of a lightweight root cause analysis method (ARCA method) – field studies at four software companies
Inf. Softw. Technol.
(2011) - et al.
A root cause analysis toolkit for TCP
Comput. Netw.
(2008) - et al.
Post-project reviews as a key project management competence
Technovation
(2008) - et al.
A system for organizational learning using cognitive maps
Omega, Int. J. Manage. Sci.
(1992) - et al.
Experiences with conducting project postmortems: reports versus stories
Softw. Process Improve. Pract.
(2005) - R.J. Latino, K.C. Latino (Eds.), Root Cause Analysis: Improving Performance for Bottom-Line Results, 6000 Broken Sound...
- T. Dingsøyr, N.B. Moe, Ø. Nytrø, Augmenting experience reports with lightweight postmortem reviews, in: PROFES’01...
- et al.
Global software development
IEEE Softw.
(2001)
Virtual retrospectives for geographically dispersed software teams
IEEE Softw.
Learning from our mistakes with defect causal analysis
IEEE Softw.
Software failure analysis for high-return process improvement decisions
Hewlett-Packard J.
Applications of defect prevention in software development
IEEE J. Sel. Areas Commun.
A case study of software process improvement during development
IEEE Trans. Softw. Eng.
Cited by (9)
Diagrams or structural lists in software project retrospectives - An experimental comparison
2015, Journal of Systems and SoftwareCitation Excerpt :Similarly, even though the use of CED has been introduced as an important part of retrospectives (Bjørnson et al., 2009), the company practices seem to favor textual notations to visualize the retrospective findings (Lehtinen et al., 2011, 2014b). Software tool support for collaborative cause–effect diagramming is also widely missing (Lehtinen et al., 2014b) and therefore using CEDs in the distributed settings is practically challenging. Thus, in terms of the tool support for modern distributed software project retrospectives, we should also determine how to visualize the outcome of RCA.
Categorization of risk factors for distributed agile projects
2015, Information and Software TechnologyCompetencies for Managing Activities in Agile Projects
2022, Global Journal of Flexible Systems ManagementGame-based Sprint retrospectives: multiple action research
2022, Empirical Software EngineeringTechnical and Managerial Difficulties in Postmortem Analysis in Software Projects
2019, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)Comparison of release engineering practices in a large mature company and a startup
2018, Empirical Software Engineering