1 Introduction

Nowadays, there is a lot of controversy around outputs vs outcomes. It seems like the industry is moving away from measuring and visualizing outputs. Every time it is more common to see roadmaps and KPIs (Key Performance Indicators) outcome driven [1, 2]. However, this fact does not mean to forget the operational part [3]. Maybe, what is needed is to know who should perform certain activities like defining the cycle time, defining stories, establishing flow efficiencies and many other usual actions inside these environments. In order to perform an effective Agile coaching [4], it is necessary to collect data on the activities carried out by the team in order to let it know the way it works. Teams do not tend to be aware of the impact generated by the way they work [5]. About who should define and establish these metrics [6], the development team is who should perform most of them [7]. It is better to be driven by the power of proof than by assumptions [8].

The value provided to customers is built going through a process, and it is that process that should be measured and analysed to see its strengths and weaknesses. We could use an analogy to explain what we mean. Imagine a person that is sick, a person that is having a health problem. The outcome we are looking for is that this person could enjoy his life back free of problems derived from the sickness. The outputs are the number of pills he or she will have to take, the number of visits to the doctor, the controls he or she will have to perform, among others.

Not all doctors know right why certain people is sick. They have to know which indicators they have to measure in order to feel certain from where the sickness might come. The doctor has to gather data and extract conclusions to be able to know why the patient is sick. Then, the doctor will be able to achieve the desired outcome for the patient by making sure he or she follows the established outputs. In our analogy, the doctor would be the Agile coach and the patient would be the team. From this, the title of the paper “Image Based Diagnosis for Agile Coaching”.

It could be very useful to have a tool to represent, compare and analyse the collected data and to facilitate decision-making. To be able to analyse the economic impact that queues cause in the organization [2], one of the most accepted terms to evaluate what is left to win due to the existence of queues is Cost of Delay. In addition to Cost of Delay, other metrics can be established in a team. Some of these metrics are Cycle time, Lead time, Flow efficiency, Cumulative flow diagram, Burndown Chart, Throughput, among others [1]. There are different software tools to collect data and graphically represent the values for the metrics established in the team. One of the tools for this kind of diagnosis is Actionable Agile. In our research, we would like to see how important is to use visual tools and how they can make behaviours change [9, 10].

This paper presents a study of the charts generated by the tool Actionable Agile that can be used by a team to represent some performance metrics. How certain decisions, like establishing a WIP (Work In Progress) limit can affect the operational dynamics of a team, or how the team can be predictable when work items are concrete and concise, are experiences included in the paper. The charts we have selected represent the metrics that can be used, on the one hand, to build team awareness and, on the other, to constantly deliver value.

The paper is organized as follows. Section 2 presents the tool Actionable Agile. Section 3 shows the charts generated by Actionable Agile that can be used by a team to interpret performance metrics. Finally, Sect. 4 opens discussion about the usefulness of the selected metrics and concludes the paper.

2 Actionable Agile: A Tool for Agile Coaching

Actionable Agile provides a set of charts that can help to understand how the dynamics of the team are behaving. That way, the user can find patterns of behaviour and try to take awareness about the actions done.

Even if the team uses a task management software such as Jira, all the information is not commonly available for free. Actionable Agile has several options to be integrated into a particular tracking tool. The three main ones are:

  1. 1.

    By buying the add-on and use it as an Atlassian add-on. This option results quite expensive as the fee is for every user of Jira instance.

  2. 2.

    By connecting Actionable Agile to Jira throughout user and password. This is a cheaper and good option since the licence for Actionable Agile is 19$ per month per user and, usually, not too many users are needed. The limitation of this option is that when connecting Actionable Agile to Jira, the tool will just migrate the different boards (Scrum or Kanban) and there is not the opportunity to customize the workflow stages.

  3. 3.

    By uploading the own dataset. This option is the one that the Agile coach participating in this paper is using. He developed a small program in Symfony that connects to Jira to generate a dataset to be uploaded to Actionable Agile.

The actions to be done in order to use Actionable Agile in a proper way are:

  1. 1.

    To specify the header of the CSV needs to have a certain format for building the data set to be used in the tool. The guidelines are available at https://actionableagile.com/format-data-file.

  2. 2.

    To indicate the Jira instance name (JQL) to be executed. The results of the execution are the different tasks to check. For instance, “all the feature that belong to a specific project” or “all the epics that belong to a specific project that were created at a certain period of time”.

  3. 3.

    To establish the number of months to export. Jira API only returns a maximum of 100 tasks. By setting the number of months to export in small batches, we can build the dataset piece by piece, and we make sure that any response does not contain more than 100 tasks.

  4. 4.

    To create the workflow stages. Some examples could be “Added to Backlog”, “In Development”, “Development Done”, among others.

  5. 5.

    To specify a column for tasks that are blocked. When a task is in this column, it cannot be moved forward. It is advisable to move a task to this column when something is blocked.

  6. 6.

    To set some parameters, such as the project code, task type, etc.

Once the configuration is ready, Actionable Agile can be executed by hitting the URL https://agile.companyname.com/actionable-agile/instance, and the file can be obtained. When uploading the file, all charts will be generated. Next section describes the charts selected by the authors.

3 Interpretation of Main Charts Provided by Actionable Agile

Actionable Agile provides a set of charts that support the analyses of different team performance metrics. According to [11], there are four factors that a metric should have in order to provide real value:

  1. 1.

    Raise warnings - a metric should warn that something happened.

  2. 2.

    Drive behaviour - it should have an impact in the people behaviour

  3. 3.

    Teach - it should help to understand how thing are going on.

  4. 4.

    Expire or recycle - it should be replaced when it is old.

In this study, authors have analysed some of the charts than can be generated by Actionable Agile. The selection was made depending on which charts could better help to represent the values for team performance metrics. That is, how the dynamics of the team are behaving. The Actionable Agile charts that are presented in the following subsections are:

  1. 1.

    Scatterplot chart: It is a type of plot or mathematical diagram using Cartesian coordinates to display values for typically two variables for a set of data. If the points are color-coded, one additional variable can be displayed.

  2. 2.

    Cumulative Flow diagram: It is an area graph that depicts the quantity of work in a given stage. It shows arrivals, time in queue, quantity in queue, and departures.

  3. 3.

    Aging Work in Progress: It helps to visualize how tasks are progressing from the initial to the final stage of the flow.

  4. 4.

    Flow Efficiency: It represents the ratio between value-adding time and the lead time (the frame between the order and delivery) required to complete a process.

  5. 5.

    Heat map: It is a graphical representation of data where the individual values contained in a matrix are represented as colours.

  6. 6.

    Throughput chart: It depicts the average number of units processed per time unit.

  7. 7.

    WIP Run: It shows the number of tasks in progress per time unit.

3.1 Scatterplot Chart

A scatterplot chart depicts the cycle time of all finished tasks. Each dot represents a task. The x-axis shows when the task was finished. The y-axis shows how long it took to be finished. This time to completion will depend on the different workflow stages selected, for instance we could select only Starting Dev, Dev Done, Start Testing, Testing Done, Task Done. If the user is only interested in seeing the data from when a development starts and not from when it was added to the backlog, it is possible to customize it by just clicking the different workflows stages. Figure 1 shows a scatterplot chart. This example does not include the time where the tasks where in the backlog, it only considers when the task started to be developed.

Fig. 1.
figure 1

Scatterplot chart

Full size image

In Fig. 1, tasks during the first period were urgent issues and, after October, there were less urgent tasks. The cycle time is relative short. Why this happened? This is something to investigate. Maybe the team dynamics changed, maybe the team spent some time paying technical debt, or maybe the team was fixing bugs so they did not become urgent issues.

Figure 2 shows another scatterplot chart with a limit of WIP tasks. It represents the results of an experiment in a Scrum team that decided to set WIP limits. The green area represents the period since the experiment of limiting WIP started. The results were quite good with reducing the cycle time of tasks.

Fig. 2.
figure 2

Scatterplot chart with WIP limit (Color figure online)

Full size image

Figure 3 shows the same experiment but with a different team. It can be appreciated that there were more tasks with less cycle time. This is because the team started to work on smaller tasks so they could be finished early.

Fig. 3.
figure 3

Scatterplot chart with WIP limit

Full size image

Different types of task can be represented as the third variable in a scatterplot chart. Figure 4 shows the cycle time of tasks grouped by task type.

Fig. 4.
figure 4

Scatterplot chart by task type

Full size image

Figure 5 depicts the tasks used for the database administrator to manipulate data. It can be seen that these tasks are finished very quickly. By checking the percentile, 85% of tasks were finished within 3 days.

Fig. 5.
figure 5

Scatterplot chart with use of percentile

Full size image

Figure 6 shows that if all team members are accurate and make a good usage of Jira, the team will be even able to use the Monte Carlo forecasting information provided by the tool.

Fig. 6.
figure 6

Monte Carlo forecasting information

Full size image

A very useful functionality of Actionable Agile is the one that allow the team to know the tasks blocked at some point. Figure 7 shows in red colour the tasks blocked.

Fig. 7.
figure 7

Scatterplot chart with blocked tasks (Color figure online)

Full size image

Why Is the Scatterplot Chart Helpful?

The scatterplot chart gives great visibility of the cycle times and the percentiles of them. For instance, it is not the same to know that the average cycle time is 10 days, that to know that 80% of cycle times is 7 days and 20% is 15 days. The information provided by the percentiles complement a lot.

3.2 Cumulative Flow Diagram

In a Cumulative Flow diagram, the x-axis shows a sequence of days and the y-axis shows the number of tasks that were in a specific workflow stage. This chart collects the number of tasks in each status at the end of each day and enables seeing how they accumulate. Figure 8 shows an example of Cumulative Flow diagram. Looking at the area in the circle, it can be seen that the red area (representing the tasks in the testing workflow stage) is quite big. Some days after that many of them were finished, this is why the blue area (representing the finished tasks) increases.

Fig. 8.
figure 8

Cumulative Flow diagram (Color figure online)

Full size image

Figure 9 shows another example of Cumulative Flow diagram. Looking again at the area in the circle, it can be seen how different workflow stages started to increase. This could mean, for instance, that the team started to accumulate too much tasks being developed, or tasks being tested, or even tasks pending to be released, which is even worst.

Fig. 9.
figure 9

Cumulative Flow diagram (Color figure online)

Full size image

Imagine that a development team has to deal with the QA that is on holidays. What can happen in this situation is that the team will start accumulating work that is pending to be tested. If the red area is the area for tasks pending to be tested, it can be seen pretty soon how this area starts accumulating, because there will be more things pending to be tested and less things finished.

Why Is the Cumulative Flow Diagram Helpful?

This diagram can show easily the health of the process. The team can observe issues such as if the backlog grows more than it finishes tasks, or if the development process suffers from testing bottlenecks.

3.3 Aging Work in Progress

This chart enables to see all the tasks that are not finished, and in which workflow stage they are waiting. Figure 10 shows an example of Aging Work in Progress chart. In this example, the following information can be extracted: (1) There are a lot of tasks in the backlog. (2) There are some tasks being tested and some tasks with testing finished to be released that have been waiting for a long time.

Fig. 10.
figure 10

Aging Work in Progress

Full size image

Why Is the Aging Work in Progress Chart Helpful?

This chart can help a product owner to give great visibility on which tasks are in progress and in which workflow stage they are waiting.

3.4 Flow Efficiency

Flow efficiency takes all the active work and non-active work of each tasks and then, it calculates the flow efficiency. For instance, if the task was in the backlog for 20 days, and then it was developed in 1 day, the flow efficiency can be calculated like this:

$$ [Work\;time/(Work\;time\; + \;Wait\;time)]\; * \;100 $$
$$ [1/[\left( {1\; + \;20} \right)]\; * \;100 = 4.8\% $$

This means that only 4.8% of the whole life cycle was actually invested on active working time. In theory, a good flow efficiency starts to be at 40%.

Why Is the Flow Efficiency Chart Helpful?

This chart, represented in Fig. 11, is quite good to represent tasks that have been waiting for ages in the backlog, as well as tasks that once they were started they were never finished because the team switched focus. Flow Efficiency can start creating awareness for the team to actually be able to finish a task with no interruptions or problems.

Fig. 11.
figure 11

Flow efficiency

Full size image

3.5 Heat Map

Heat map uses colours to show the most common workflow stages. In the example in Fig. 12, it can be seen how the column for tasks Added_To_Backlog is the most affected. This is not strange; teams usually tend to create more tasks than the ones that is able to process.

Fig. 12.
figure 12

Heat map (Color figure online)

Full size image

Why Is the Heat Map Chart Helpful?

This chart can spot issues such as QA bottlenecks or problems with releases. For instance, if the team is not able to release something as soon as it is finished, the workflow stage that represents this situation will start changing the colour.

3.6 Throughput Chart

In a Throughput chart the x-axis shows the time and the y-axis shows how many tasks were finished. In the example in Fig. 13, the three circles show three days in which the team finished many tasks. This chart can show if sprints are waterfall sprints. A waterfall sprint is a sprint in which all tasks are closed during the last days.

Fig. 13.
figure 13

Throughput chart

Full size image

Using this last example, we can see that this usually happens in teams where there is a QA and the QA acts as a manual tester. Having one QA for every four or five developers will affect the process; and when it happens it will be reflected in the Throughput chart, because the team will finish all their tasks during the last days of the sprint.

Why Is the Throughput Chart Helpful?

It can show the waterfall pattern, that is how often the team finishes tasks. Does it finish on batches of sprints, or does it daily?

3.7 WIP Run

WIP Run chart shows how many tasks are in progress (y-axis) every day (x-axis). If the team has defined a WIP limit, it will be able to see if this WIP limit is being respected or not. It is very useful to detect strange patterns, such as the ones in red colour in Fig. 14. In these cases, many tasks are finished and the WIP count dropped.

Fig. 14.
figure 14

WIP Run (Color figure online)

Full size image

This chart is more powerful than it seems. We believe it is quite telling. Using this chart during a retrospective, the team can see how many activities has been working on at the same time during the sprint.

Why Is the WIP Run Chart Helpful?

It can show the ability of the team to not accumulate too much work in progress.

4 Discussion and Conclusion

The best thing to start with is by understanding what these metrics are and what they mean. This is the best position to connect all the metrics with the value they provide. Definitely, a shorter cycle time or a higher throughput does not really mean more value added. It just means that the operational process might be better than before.

As it can represent a lot of new information at the beginning, the recommendation of the authors is to start by the Scatterplot chart and the Cumulative Flow diagram. Using them together, a quite nice view of the health of the operational part can be obtained. Moreover, silos within the process and struggles like dependencies could be discovered.

It is important to remark that this tool is not built to look at people but to have teams to succeed and to move forward together. We do not recommend using Actionable Agile for individual performance reviews. Data are great, but they can also be dangerous if they measure just one part of the system and not the system as a whole. The team will need time to understand these measures and why to apply them. It is not convenient to bomb the team up with a lot of data at the beginning. The team should be asked about what they would like to see.

Sometimes it seems like that in Agile the topic of using data (beyond story points or velocity) it is a bit of a taboo. Agile looks at the human part as well. Agile looks for teams performing and happy at the same time. Depending on the state of the team data can be convenient or not. If a team is suffering because the members do not have a good relationship, data won’t be useful at this point. But soon as the team starts performing, data will be needed.

We really agree with the sentence “What cannot be seen does not exist”. No matter how much a team complains because they have dependencies, or because they lack expertise on a topic. If it is not visible, then the problem does not exist. Is the team suffering because of dependencies? Let’s make them visible! Is the team suffering because they lack skills? Let’s make the cycle times visible! With this information, management and team leads will be worried about solving these issues.

All the information collected from the tool by analysing the charts it produces is very useful for an Agile coach to make decisions together with the teams. The best situation to help in a decision-making is to know if there was an improvement or not from past situations, and to visualize how the team improved their operational process. At the end, an Agile coach will succeed if the teams he or she works succeed as well.