1 Introduction

Organizations are searching for ways to achieve the defined business goals and overcome software development barriers. Agile methods are an option many organizations have chosen to try to reach success [1,2,3,4]. Agile methods adoption has been growing [4] and it creates the need to guide organizations through the transformation process [2, 5, 6]. Researches show that it is hard to adopt out of the box agile methods and that there no unique path to success [2, 5,6,7,8].

During agile transformation processes (ATPs), organizations go through important changes that have deep impacts in multiple aspects: culture, hierarchy, management, environment and people [8, 9]. Understanding the challenges or success factors for an agile transformation helps to prepare the organization and the people involved and increase chances of success. Agile transformation projects that do not consider the challenges to be faced along the way do not bring positive results to the organization. It can also create rejection from the professionals involved in the ATP. Understanding the impacts of the changes in the organizational environment is an important step in the adoption process [9].

The literature is rich in agile success factors involved in an ATP [8, 10,11,12,13] but there is still no direct guidelines of how to use these success factors in specific organizational contexts. The usage of success factors as a tool to help in the agile transformation initiatives can start with the awareness of how organizations and agile practitioners implement these success factors to create a fertile environment for ATPs and how hard this implementation can be.

This research intends to investigate the level of difficulty of agile success factors implementation according to practitioners point of view and to create a ranking that can be used as a reference for organizations in any stage of their ATPs. We proposed an assessment as a tool to provide awareness of the current status of agile success factors in the organization and the gaps to get to the target state defined by organization leadership. Thus, we used this assessment to conduct a survey to gather information from agile practitioners about success factor implementation in their organizations. Using the survey results as input, we applied the Rasch algorithm [14] to create a success factors implementation difficulty ranking identifying which are the hardest success factors to implement and which are the easiest ones.

The remainder of the paper is organized as follows. Section 2 provides the technical background on ATPs and success factors while Sect. 3 describes the methodology used in the research. The results of the survey and the findings are discussed in Sect. 4 and related work is summarized in Sect. 2.3. Section 5 outlines the conclusions and future work opportunities.

2 Background

This section reviews two key concepts: agile transformation process (ATP) and agile transformation success factors groups. It also provides a review of the related work found in the literature.

2.1 Agile Transformation Process (ATP)

Agile transformation process (ATP) is the process of transforming an organization into agile [15]. This process impacts all areas of the organizations. People are required to change their mindset and behavior [16, 17], they need to modify the way they work as a team. Processes are affected and need to be adjusted, management style and attitudes are affected as well [15].

It is important that organizations understand the reach of the changes involved in ATP. It helps to create the awareness needed for this type of initiative, prepares the team for the upcoming challenges and increases the chances of achieving the proposed goals [10, 15]. Further, organizations need to control effort and costs associated with ATP [18].

2.2 Agile Transformation Success Factors Groups

Agile transformation success factors have been studied by researchers to point paths to allow organizations to have control over their adoption process and to be able to deal with known barriers and challenges during the implementation [18].

Some researchers refer to the success factors as challenges for agile adoption (Nerur et al. [9], Conboy et al. [17] and Gregory et al. [11]) but it is basically a different view of the same factor. Success factors are the basis of the agile transformation success factors assessment since the evaluation occurs at the success factor level. During the assessment definition, we found a need for a higher level of evaluation to allow a consolidated management view of the success factors. This view is provided by the success factor groups defined in this section.

We identified multiple success factors in the literature and aggregated them according to their concepts into terms and then into a set of groups. There were multiple references to the same concepts in different articles and we used the terms we considered more representative of the concepts to aggregate all the references found. The six groups were proposed organizing the terms according to the areas of the company affected by them are summarized in Table 1: customer, management, organization, process, team and tools.

Table 1. Agile transformation success factors and groups proposed by the authors.
Full size table

Customer Group. Customer involvement and relationship are cited by multiple researchers as a relevant factor in agile transformation [9, 10, 12, 20]. Customers should participate of the projects in a regular basis, establish effective communication, be able to make decisions and provide feedback [9, 12, 19].

Management Group. Management plays an important role in the ATP supporting the team’s empowerment and creating conditions for changes to take place [15, 18]. Management support is a prerequisite for agile adoption [15, 16] because people need to see their buy-in to feel involved, to give the expected importance to the adoption initiative and to understand that adjustments can be made to the organization and its structure to support the new proposed paradigms [10].

Organization Group. The organization is one of the main focuses of the ATP. Multiple deep changes happen at this level, covering aspects driving the professionals behavior in the company such as culture and values [9]. As software development is a knowledge based activity, learning and flexibility are important attributes for the organizations involved with ATPs.

Process Group. The process group covers the adequacy of the process to the organizational context. In ATPs, it is common to change the development process to adopt an iterative, incremental, secure, transparent and people focused process. However, it requires significant effort and a careful analysis to create a process that makes sense for the reality of the teams [9, 17].

Team Group. Agile environments are all about people. It requires people to be involved and to participate in agile adoption since the beginning of the ATP [15]. People buy-in is a facilitator of the transformation process [10, 16], involved people can help in the transformation and attract other people to be part of it [15].

Tools Group. Here we see an interesting conflict between agile values and reality. We understand that in a people-centric approach such as agile methods the individuals and interactions are highly valuable. However, in the practice, tools are important to allow teams to transition their work procedures in a more natural and productive fashion [9, 13, 19].

2.3 Related Work

Taylor et al. [10] proposed a minimally intrusive risk assessment to prepare small companies to adopt agile methods. The articles express concerns about the capability of small companies to survive to a failed agile adoption attempt and proposes the assessment to point directions if the organization should pursue the agile adoption or use a traditional approach for software development. This assessment can provide a direction on agile adoption per project basis. The authors used the agility/discipline assessment (ADA) developed by Boehm and Tuner [25] and they applied it to six organizations proposing adaptations to each of the contexts. The areas contemplated by the assessment were criticality, personnel, dynamism, culture, client involvement and team distribution.

Gandomani et al. [13] provided a comparison between traditional and agile software development methods and the challenges of transitioning to agile methods. They classify the challenges for agile transformation into four categories: organization and management; people; process; technology and tools. Their work also highlighted that the organization as whole is affected by this process. A strong message from their study is that all members should be involved to deal with such challenges.

Chow and Cao [12] investigated the critical success factors in agile methods adoption and proposed groups to classify critical success factors and created a model to validate how the critical success factors affects the quality, cost, scope and time of agile software development projects. The groups of factors were defined as organizational, people, process, technical and project. They conducted a survey to validate hypothesis for each of the success factors and analyzed the correlation between success factors and project success to establish critical success factors. The critical success factors according to their results are: delivery strategy, agile software engineering techniques and high qualified team.

3 Methodology

In order to reach the goals of this study, we planned the data collection and analysis starting with an assessment called “Agile Success Factors Assessment”. It was used as a questionnaire to gather data for this research. The second step was to apply the assessment as a survey to agile practitioners in an event named “Agile Trends” in the city of Belo Horizonte, Brazil on September 10th, 2016. After the event, we also published the questionnaire as a on-line survey form to be responded by agile practitioners. The third step was to apply the Rasch algorithm [14] to generate a success factors implementation difficulty ranking.

3.1 Agile Success Factors Assessment

The assessment was created using the proposed success factors and groups. We formulated phrases to represent each of the success factors and to allow their evaluation during the assessment. The main objective of the assessment is to be used as a tool to provide awareness of how the organization evaluates itself, defines its goals for the ATP and which should be the next steps in the process considering the feedback from people in the organization regarding their current state.

The assessment is composed by 23 phrases representing the success factors. The assessment has been previously tested by agile experts. In this research, the phrases were evaluated by the respondents to measure the level of implementation of that success factor for that organization.

3.2 Survey

We conducted a survey in an agile event in the city of Belo Horizonte, Brazil and we published an on-line version of the survey as well. The survey consisted of two parts. Part One was where respondents identified their years of work experience, years of agile-related experience and the position they hold in the company. The demographic questions did not identify any of the respondents or their organizations. Part Two was composed of the Agile Success Factors Assessment and two scales for the respondents to inform their current level of implementation of each success factor and their target or ideal level of implementation of the success factor based on their organizational context. The level of implementation was measured using a 5-point ordinal Likert scale. We received a total of 457 responses of which 329 questionnaires were considered valid to be analyzed.

Respondents Profile. The most common position of the respondents was “project manager”. The second most common position for the respondents was “analyst” followed by “developer”. Table 2 summarizes the distribution of positions among the respondents. Table 3 shows that 33.7% of the respondents declared to have 16 or more years of work experience. 250 respondents (76.0% of the total) have at least eight years of work experience. A high number of the respondents (41.6%) declared to have from one to four years of experience using agile methods and practices while 5.6% of the respondents have 11 years or more of this type of experience. 41.7% of the respondents have five or more years of experience with agile methods and practices as summarized in Table 4.

Table 2. Distribution of the positions of the respondents.
Full size table
Table 3. Distribution of the years of work experience of the respondents.
Full size table
Table 4. Distribution of the years of experience of the respondents with agile methods and practices.
Full size table

Data Analysis. Based on the success factors’ current state and the target state gathered in the survey, we applied the Rasch algorithm [14] to generate the success factors implementation difficulty rankings. This approach has been used by Lahrmann et al. [14] to create maturity models in design science research. In this research, since the respondents came from different organizations, we used the median of their target state values for the success factors to create the desired results for the Rasch analysis. This was used in order to provide a consistent value across all the organizations evaluated as proposed by Lahrmann et al. [14].

4 Results

The intention of this research was to understand how agile practitioners perceive the level of difficulty of agile success factors implementation and create a ranking that can be used as a reference for organizations in any stage of their ATP. After the initial analysis of the data gathered in the survey, we used the agile success factors assessment to calculate different success factors rankings based on four groups of respondents: all respondents (Group 1), respondents with eight or more years of work experience (Group 2), respondents with five or more years of agile methods experience (Group 3) and respondents with eight or more years of work experience and five or more years of agile methods experience (Group 4).

We used the data gathered segmented by group to generate a success factors implementation difficulty ranking for each of the groups. Group 1 was composed of all 329 respondents and Group 2 of 250 respondents. 137 respondents were part of Group 3 and 131 respondents composed Group 4. In this setup, one respondent could be part of all groups based on his profile information. The results for each group were compared to draw the conclusions. The intention of the groups analysis was to compare how groups ranked the success factors and check the impact of general work experience and agile methods experience on the rankings.

4.1 Group 1

We executed the Rasch algorithm using the Winsteps software [21] on the data collected for Group 1 to obtain the item calibration. The item calibration resulting of the Rasch algorithm classifies the success factors in the assessment according to their implementation difficulty. The hardest to implement success factors are at the top of the ranking. The ranking and the Rasch algorithm results for Group 1 are summarized in Table 5.

Table 5. Success factors implementation difficulty ranking and Rasch algorithm results for Group 1.
Full size table

In order to validate the results from the Rasch algorithm, we used to the recommendations provided by Winsteps documentation [21] and the guidelines used by Lahrmann et al. [14]. The fit statistics values (Infit and Outfit) are around 1.00 satisfying the fit expectations and validating the results. All the execution and validation steps described for Group 1 were also used for the other groups.

Based on the results, the success factors that would be the hardest to implement are: training; measurement model; coaching and mentoring; changes in mindset of project managers and decentralized decision making; and new mindset and roles. Furthermore, the results point out the success factors that would be the easiest to implement: management buy-in; technical activities and skills; incentives and motivation to adopt agile methods; knowledge sharing; and team involvement.

4.2 Groups 2 and 3

The respondents with eight or more years of work experience composed Group 2 independently of their experience with agile methods and practices. These respondents are the ones with a considerable amount of work experience. Group 2’s ranking shows that the easiest to implement success factors are: management buy-in; technical activities and skills; knowledge sharing; incentives and motivation to adopt agile methods; and self-organized teams. The success factors considered the hardest to implement by Group 2 were: measurement model; coaching and mentoring; training; new mindset and roles; and changes in mindset of project managers.

Group 3’s respondents had five or more years of experience with agile methods and they were considered experienced practitioners. We did not consider their work experience in this analysis. The top five success factors (the hardest to implement) for this group were: measurement model; training; changes in mindset of project managers; coaching and mentoring; and new mindset and roles. The results for Group 3 presented the following success factors as the easiest to implement: incentives and motivation to adopt agile methods; management buy-in; technical activities and skills; knowledge sharing; and self-organized teams.

4.3 Group 4

Group 4 is a restrictive group being the intersection of groups 2 and 3. The 131 respondents part of this group (39.8% of the total) have work and agile methods experience to be considered experts practitioners in this study. The success factors implementation difficulty ranking for Group 4 is presented in Table 6.

The hardest to implement success factors for Group 4 are: measurement model; changes in mindset of project managers; training; coaching and mentoring; and new mindset and roles. At the bottom of Group 4’s ranking, as the easiest to implement success factors we have: incentives and motivation to adopt agile methods; management buy-in; technical activities and skills; knowledge sharing; and self-organized teams. We considered Group 4’s ranking as the one to be used as reference for organizations during their ATPs because it represents the view of the expert practitioners on the success factors implementation difficulty levels.

Table 6. Success factors implementation difficulty ranking and Rasch algorithm results for Group 4.
Full size table
Table 7. Spearman’s rank correlation coefficient results for the groups of respondents with significance at the 0.01 level.
Full size table

4.4 Correlation Between Rankings

After we obtained the groups’ rankings, we proceeded to the correlation analysis in order to understand the correlation between the rankings of the different groups. We used the Spearman’s rank correlation coefficient [22, 23] to identify the correlation between the groups’ rankings. The Spearman’s rank correlation coefficient was calculated for all the rankings using the SPSS tool and the results are summarized in Table 7.

According to Butler [24], the critical or minimal correlation value to be accepted for the Spearman’s coefficient considering the 23 observations (success factors) used in this research and applying a significance level of 0.01 in a two-tailed analysis is 0.532. All the correlation coefficients presented in Table 7 have values higher than the critical correlation value of 0.532 and they are also considered very strong correlations since the values are higher than 0.900 [22]. All the correlation coefficients were calculated considering a significance level of 0.01 that means there is a 1% chance that the relationship found happened by chance.

The strongest correlation is observed between groups 3 and 4 with a value of 0.994. This is explained by the respondents being almost the same. Group 3 has only 6 respondents that are not part of Group 4. The most relevant result in this correlation analysis is the correlation coefficient between groups 1 and 4. It represents a very high correlation level with a value of 0.957.

4.5 Discussion

In this section we will discuss the success factors found in the top 5 and bottom 5 positions of the rankings for groups 1 and 4. The correlation between these groups will also be examined. We are considering only these groups because they reflect the general sample of all respondents (Group 1) and the expert practitioners (Group 4).

Among the top 5 of the rankings we have the hardest to implement success factors (see Tables 5 and 6). Training is ranked 1\(^{st}\) for Group 1 and 3\(^{rd}\) for Group 4. Gandomani et al. [16] shows that lack of training is a challenge for ATPs and that training can be used to correct wrong mindsets. Chan and Thong [20] state that training can improve the person’s learning about agile methods and impact the knowledge transfer to the practice. Training implementation can a challenge in organizations when you consider tight budgets, small teams and the team’s work load.

As the hardest to implement success factor for Group 4 and ranked 3\(^{rd}\) for Group 1, measurement model is a challenge in ATPs [13, 19]. The lack of measurement practices can represent a limitation for the organization to understand the progress of the initiative and it will not allow comparison with the previous state of the organization. Coaching and mentoring ranked 4\(^{th}\) in both rankings. ATPs involve multiple human factors and the people are the center of these processes [13, 16]. That makes the coach role an important role in the ATP. The coach should be involved in the planning phase to provide awareness of the risks involved [15]. The challenge in using coaches is related to economic constraints and also to the acceptance of the coach by the teams.

Changes in mindset of project managers ranked 4\(^{th}\) for Group 1 and 2\(^{nd}\) for Group 4. The role of project manager needs to change from planner and controller to facilitator [13]. This is a considerable challenge for the formal project managers used to traditional software development and project management approaches. Their role should shift to the role of the team’s facilitator of the collaboration, creativity and groups decisions.

New mindset and roles ranked 4\(^{th}\) for Group 1 and 2\(^{nd}\) for Group 4. ATPs require not only cultural changes but operational and technical changes that at the end will demand a change in the way people think [11]. Culture and mindset are hardest aspects to be changed in an ATP [19]. The acceptance of a new mindset and a new set of roles involves the participation of all levels of the organization to engage people in the ATP, to provide awareness and knowledge about agile methods, to create a secure environment and to encourage people to embrace the new way of thinking and working. At the bottom of the rankings we have six success factors among groups 1 and 4: management buy-in; team involvement; knowledge sharing; incentives and motivation to adopt agile methods; technical activities and skills; self-organized teams (see Tables 5 and 6).

Management buy-in is one of the most cited success factors in the literature [9, 11, 12, 15, 16, 20]. Management buy-in is important to the ATP since it provides access to resources (budget, time, people, reach within the organization) and it shows to all people involved in the ATP that this is an important initiative. Team involvement was also considerate an easy to implement success factor. This shows that the respondents considered the team involvement an easy to implement success factor or that this success factor is already implemented in their organizations. That can be based in the perception the practitioners have of the benefits of agile adoption such as increased team motivation and increased team productivity [4].

Knowledge sharing is a practice already adopted by multiple companies and not exclusive of agile methods. This could explain why it is considered an easy to implement success factor. There is also the fact that knowledge sharing is a practice valued by software organizations or departments considering the high levels of complexity of existing applications and the technologies used to build these applications. Incentives and motivation to adopt agile methods play an important role engaging people in ATPs [15,16,17, 20]. If the organization is able to provide the correct level of incentives to the people that needs to participate in ATPs, it can increase the chances of getting motivated teams and of achieving the established goals.

The investment in technical skills and activities is considered a common task for technology related professionals. These professionals are always learning new technologies and researching new ways of doing their work because of the constant evolution of the technologies available in the market. Self-organized teams ranked as an easy to implement success factor intrigued us. Self-organized teams are related to decision making, learning new skills as the person plays different roles and freedom to operate as the team wishes to get to their goals done [13, 17, 20]. We would need further investigation of this success factor to make sure respondents understood this concept and what it means for them in their organizations.

The correlation values between all groups are higher than 0.900 which is considered very high [22]. It is interesting to notice that even the correlation between the most divergent groups in terms of respondents profile (Groups 1 and 4) is very high. This indicates that success factors with a high rank for Group 1 would also have a high rank for Group 4 and vice-versa. The correlation between groups 1 and 4 also shows that the understanding of agile success factors among practitioners varies based on their experiences and the positions of the success factors between groups’ rankings varies based on experience as well but both groups of practitioners have similar success factors as challenging to implement and consider other similar success factors easy to implement.

The assessment proposed by Taylor et al. [10] has similarities to our work: it is used to evaluate the risks of agile adoption, the goal is to involve the team members in the discussion around agile adoption and it requires a minimum overhead. Their work focuses on the choice between agile and traditional approaches at the project level while our work is focused on the preparation for an ATP and guidance for the agile success factors implementation at the organization level. Their assessment considers different aspects from the ones considered in our assessment. Our research also generated a success factors implementation difficulty ranking to be used by organizations in their ATPs.

The work of Gandomani et al. [13] work is similar to ours up to the point of the proposed success factors groups. The groups are similar between the two researches but we have aggregated terms from different references. However, their work stops at this point while, in our research, the success factors groups are used as basis for the agile transformation success factors assessment definition. We understand that our research has a practice oriented goal with the usage of the assessment and the generation of the success factors implementation difficulty ranking to serve as a road map for the ATP. Chow and Cao’s work [12] has similarities with this research on the usage of the success factors groups approach, on the survey to gather data for the research and on the usage of statistical methods to analyze the data gathered. The differences are that they proposed a model to associate success factors to software development projects aspects while we used a success factor assessment and proposed the success factors implementation difficulty ranking targeting the organization as a whole and not a specific project.

The limitation of this research is that the data might not be a generalized random sample of agile practitioners. The sample population of this research might not be representative of the agile practitioners’ community in general. The survey gathered anonymous data preventing data validation. The idea of the success factors implementation difficulty ranking is to represent the findings for the analyzed sample population of expert practitioners. However, we are aware that the organizational context and the people involved in the process make it rather unique and our findings can be used as a reference but might not be applicable to all cases.

5 Conclusions

In this study, we used an agile transformation success factors assessment to gather data about agile success factor implementation difficulty in a survey among agile practitioners. The analysis of the data was done using four groups of respondents: all respondents (Group 1), respondents with eight or more years of work experience (Group 2), respondents with five or more years of agile methods experience (Group 3) and respondents with eight or more years of work experience and five or more years of agile methods experience (Group 4). Group 4 was considered the group of expert practitioners. The results generated a success factors implementation difficulty ranking for each group showing which are the hardest and easiest success factors to be implemented by organizations.

The rankings presented in this study list the success factors according to the difficulty to implement them in the organization considering its context (culture, reality, goals, hierarchy). For the group of expert practitioners, the hardest to implement success factors were: measurement model; changes in mindset of project managers; training; coaching and mentoring; and new mindset and roles. Meanwhile, the easiest to implement success factors were: incentives and motivation to adopt agile methods; management buy-in; technical activities and skills; knowledge sharing; and self-organized teams.

The correlation coefficient between the rankings also showed very high correlation among all the rankings. This means that success factors with a high rank for one rankings would also have a high rank for another ranking and vice-versa. This is an indication that both expert practitioners and non-experienced practitioners face similar challenges when implementing the success factors with some variation according the their organizational context.

The contributions of this research are the success factors implementation difficulty ranking and the correlation between the rankings. The ranking can be used as a tool to help organizations to understand the current state of agile success factors in the organization based on their team members view and to prepare for the agile transformation process. The correlation findings can be used to provide a further view of how expert and non-experienced practitioners look at the challenges in ATPs. Future work would involve using the success factors implementation difficulty ranking and the agile transformation success factors assessment in organizations to provide a diagnostic of their current state regarding success factors and use it as an input for the ATP.