Keywords

1 Introduction

A large number of information system (IS) implementation projects still fail to reach their objectives [1]. Many attempts have been made to develop an understanding of the critical success factors for IS projects. However, the project success rate has not improved significantly in the past [1], despite the comprehensive knowledge gathered by practitioners and researchers. Researchers argue that project success is a multidimensional construct with different dimensions of success [2]. Each dimension can be of different importance in different projects. This indicates that there is a multitude of possible configurations for successful or unsuccessful projects and therefore an incongruence of IS project success as the phenomenon of interest and the analysis methods commonly used for its evaluation. Our interpretation of this issue in research is in line with previous efforts in general project success research [3]. This enables us to go beyond the identification of factors for project survival [4] to the evaluation of constellations of relative project success. It is the basic premise of our configurational approach that a configuration consists of a constellation of characteristics, which are conceptually distinct and commonly occur together [5]. Generally, we view the different characteristics of an IS project as parts of a project’s configuration. Thus, we propose a remedy for the incongruence of research method and project success as the phenomenon of interest. We use the set-theoretic method of Qualitative Comparative Analysis (QCA) [6], specifically, the fsQCA method, which is generally established in other fields such as political science [7].

Many different characteristics of IS projects’ can have a substantial effect on IS project success. Thus, a multitude of different configurations or characteristics can be related with a successful IS project. However, user involvement and participation (UIP) have been identified as some of the most important factors to ensure overall IS project success [1]. In particular, Bano and Zowghi [8] suggested to analyze the level and degree of UIP required to achieve project success in different project phases, as they did not find any such guideline in the literature. Furthermore, there is still a lack of knowledge about the appropriate timing for UIP, even though it has mostly been suggested that UIP is important in requirements analysis or during testing to positively influence project success [8]. Inspired by this, we focus our research on factors of UIP related to project success. We measure project success in terms of end-users’ perceived usability, as UIP normally serves the purpose of improving the user interface and work processes of a system [1]. The analysis of the appropriate point in time, the kind of UIP, users’ involvement and motivation, and the types of involved users are our main research interests. Based on these insights, we propose the following research question: How are different forms of user involvement and participation in IS implementation projects related with IS project success?

We answer this research question with an analysis of multiple-case studies of the relationship of project success and different forms of UIP in 16 different IS implementation projects. The cases are completed IS projects with a user interface for commercial users. We employ a mixed methods research approach combining qualitative and quantitative research methods to answer the research question, as such an approach is required to study the complex and multifaceted relationship between UIP and project success [8, 9]. We contribute to the IS project research by presenting additional evidence for the most effective point in time for end users’ involvement and participation in an IS project. In our analysis, we are further highlighting that successful UIP is not only about when to best involve users, but also about showing what kind of involvement works best when and what pattern of relative configurations of such involvement can contribute to further improve IT project success. Therefore, our study advances previous IS project success research by moving beyond the often anecdotal evidence and experience reports by practitioners about the best point in time for user participation in IS implementation projects. We are able to show the applicability of a configurational approach and specifically fsQCA for IS project analysis. Our research also benefits practitioners because it gives them an indication to focus their resources on aspects of a project for which intensive UIP is crucial instead of using it in all phases of the project.

2 Theoretical Background

Generally, IS project success has been measured based on the dimension of IS product success [10]. It is possible to measure IS product success based on indirect indicators such as, for example, user satisfaction [11]. Usability is a very important software product characteristic [12] and can help to achieve user satisfaction. It is a higher design objective and an attribute of software quality [13]. The definitions of usability are often bound to the measurement method [13]. Hence, numerous definitions for usability exist [14]. We will adopt the process-oriented ISO 9241 standard as the definition for the purpose of this research [15]. We chose the ISO standard because it is one of the most prominent and widely used definitions. It does not focus on the characteristics required for the user interface but rather on the procedures of a system. Hence, we decided to use the values for the System Usability Scale (SUS) [16] as an indication for project success. Especially, because it is very established and allows for benchmarking based on values gathered in other projects [17, 18].

User participation and involvement (UIP) have been identified as positively related to IS project success in previous research [9], although earlier literature reviews have produced conflicting results [19]. Harris and Weistroffer [20] name several advantages of UIP including preventing the adoption of unneeded, costly features and an improved quality of IS due to requirements that are more precise. The role of users in the provision of the tacit process and work context knowledge, which is necessary to evaluate requirements, is also highlighted by other researchers [8]. If users participate in a project, they are also more likely to claim ownership of a system [21]. Based on such an understanding, user participation can be seen as an antecedent of user involvement [22]. McGill and Kobas [23] have shown that such participating users perceive a new system as more useful and will have a more positive attitude towards a project. In part, this can be attributed to the fact that users develop a more realistic expectation regarding the features and connected capabilities of the IS [24].

The terms user involvement and user participation have often been used synonymously by researchers [8]. This has happened despite of early efforts to develop distinctive definitions for these two aspects of project management. For instance, Barki and Hartwick [25] introduced the following definition: User involvement is a “subjective psychological state of the individual, defined as the importance and personal relevance of a system to a user”, while user participation is a “set of behaviors and activities [that] users perform” [26, p. 53]. We are going to follow this definition in this paper. Users can therefore be involved in an IS project without participating and performing any activities on their part [8]. Discussing UIP in more detail also requires a definition of the actual users of an IS. Broadly defined, users are all non-technical employees of an organization who are affected by the IS [27]. Thus, we define a user as someone with direct interaction with the system or as someone who is going to have it in the future.

With regard to the ideal point in time for UIP, Bano and Zowghi [8] state that it is widely believed that user participation in early project phases is most effective. However, it is not enough to just involve users in any project stage, instead this has to be done in an appropriate manner. Bano and Zowghi [8] also argue that the different project phases require different types and levels of UIP for an ideal contribution to project success. Considering different phases in more detail, in requirements analysis UIP helps to better understand the requirements of the users [8]. In development and customization for an IS implementation project it helps that the user requirements are purposefully transformed into technical solutions [27, 28]. Moreover, user participation in the testing phase can ensure that the user requirements are actually fulfilled by the developed system. End-user training helps users to learn how to use the system and therefore contributes to project success [29]. Based on the aforementioned insights, we classify the phases for user participation (see Fig. 1) [8]. Nonetheless, in all phases there can also be “token” user participation that does not really influence overall system success, but is rather a half-hearted measure to gather input from users [28]. The point in time, the degree and level of UIP can also influence project success [8].

Fig. 1.
figure 1

Research framework [9, 31]

Full size image

Damodaran [30] (see Fig. 1) developed an approach, which we adapted for the assessment of user participation over the course of a project. There are three levels of user participation of which the latter is the most extensive form: informative, consultative, and participative. The informative form of user participation means that users provide information to and receive information from the project team. That implies that users affect the project indirectly, but do not actively participate. If users have a consultative role, they comment on predefined services or a range of facilities. For instance, they comment different types of artifacts developed during the project [8]. In a participative role users influence decisions that are related to the whole system [30]. In such a setup, at least some users can be understood as part of the project team [8]. The level of user participation and the types of participating users is an additional characteristic of the particular configuration of a case.

3 Research Methodology

In this chapter, we present our approach of two analysis steps. First, we analyzed the individual cases. This ensured a thorough understanding of the cases as individual configurations, which is necessary for a successful QCA approach [31]. The data for the individual cases was gathered individually by some of the co-authors. Second, we present our cross-case analysis approach based on configurational theory. This second step is an integrated analysis of the data gathered by some co-authors in the first round.

3.1 Data Collection

While we gather data on the independent variables using semi-structured interviews with project members, we use an online survey among software users to measure the perceived product success based on usability.

Selection of the Cases.

We selected cases with substantial differences in the aspects under investigation [32]. For instance, cases differed in type and degree of UIP as well as in the complexity of the tasks represented in the software concerned in the project. This ensures some generalizability of our research results. Furthermore, we needed to ensure the comparability of the projects. In general, we chose implementation projects of standard software and projects in which software was developed and implemented. Thereby, we considered implementation projects of completely new IS as well as projects of substitutions of an old IS. Moreover, it was a prerequisite that the considered software has a user interface and that business users work with it. Considering the context of the cases, we selected projects taking place in companies in Germany as well as in public organizations. Furthermore, we got the opportunity to conduct interviews in Columbia because of the contacts of one of the authors. We gained access to the different projects via personal connections of individual researchers and cold calling efforts. Lastly, as different effects of the project configuration on project success are subject to our analysis, we made sure to gather users’ evaluation in relation to the circumstances of the project and not to other factors that occurred afterwards. Therefore, we only selected IS which still were in the shakedown phase [33]. This phase “refers to the period from ‘going live’ until ‘routine use’ has been achieved and can typically last anywhere from 6 months to a year” [34].

Interviews.

For each case, some co-authors conducted two interviews: one with the IT project manager and one with a project member, ideally a user of the software. We chose this approach to obtain a certain breadth of opinions [35]. We conducted semi-structured interviews [35]. As recommended by Myers [35] we allowed interviewees to tell their own stories but created an interview guide before the interviews series. This allowed us to structure the interviews and to ensure that we asked all necessary questions and collected all the information required for the multi-case analysis. We asked interviewees questions about project characteristics, such as the introduced software and the conditions of the project, and the UIP characteristics of the case. On-site visits allowed us to conduct face-to-face interviews. We interviewed the project manager separately from the other project member to ensure that the two interviewees did not influence each other. If it was not possible to have a face-to-face interview, we conducted them either via phone or via video-call. We recorded and transcribed the interviews, if interviewees agreed. Otherwise, we took notes during the interviews to enable coding in a later stage.

Online-Survey.

We used an online-survey among users of the particular software for each of the cases to measure the level of perceived usability. We chose a web-based survey form as it requires less effort of the respondent than other survey forms [36]. We used the System Usability Scale (SUS) developed by Brooke (1996) with a 5-Point-Likert-Scale to measure the usability of the software implemented in the particular projects [17, 18]. We chose it for three reasons. First, it is flexible enough to evaluate a wide range of different interface technologies [18] which is important to ensure that the survey works for all considered projects. Second, it is quick and easy to use for both participants and administrators [18]. Third, the result of the SUS is a single score that can be easily understood by many people and which is comparable to the results of other published studies and examined software systems [18, 37]. One researcher translated all statements into German, a second person translated them back into English, and a third person confirmed parity with the original statements. If this was not case, we revised the translations. The same process was executed for the Spanish survey.

3.2 Data Analysis

As suggested by Eisenhardt [38], we employed a two-stage approach comprising a within-case analysis and a cross-case analysis to analyze the case study data. First, the within-case analysis is used to develop a case study write-up for each of the cases [38]. Second, we conducted a cross-case analysis with the fsQCA approach [6, 26] to find patterns across cases and thus to identify generalizable findings. We chose fsQCA as a data analysis technique because it allows to determine the logical conclusions that are supported by a given data set.

Within-Case Analysis.

We analyzed the considered cases separately to gather detailed information about the projects. We used a coding strategy to reduce, organize, and classify the data [35]. ATLAS.TI was our software for the coding and analysis process of qualitative data because it is widely used and well-established for qualitative analysis [39]. We followed three not consecutive coding steps: open coding, axial coding, and selective coding [40]. When we analyzed the survey data, we compared the results of the online survey for the SUS with published averages. For instance, Sauro [41] compared the results of 446 studies and respectively more than 5,000 observations and states that the average SUS score is 68 (similarly 67.6 for business to business applications) whereby the bottom third has a score up to 60 and the upper third a score higher than 72. You find a brief overview of the cases in Table 1.

Table 1. Overview of the selected cases.
Full size table

Cross-Case Analysis.

After analyzing each case separately, we conducted a cross-case analysis using the fsQCA method. We use the work of Schneider and Wagemann [7, 31] as well as of Thiem and Duşa [42] as guidelines for the application of fsQCA. The argument for choosing fsQCA is especially based on the categorization of the outcome [31] that it allows. We use the software called fs/QCA, Version 3.0 [43] for the initial analysis and replicated our analysis with a QCA-package for R [42, 44]. Furthermore, we also used the QCA-package for R in the data calibration process, which is necessary when using fsQCA. We used the framework for the assessment of user participation, that we developed based on Damodaran’s [30] forms of user participation, for the assessment of user participation across the different projects. Thus, it served as a qualitative anchor during set calibration by direct assignment [31]. We identified the three main phases for user participation from our model in all projects. In light of the available data, fsQCA confines us to a reasonable number of conditions [31].

Requirements analysis and design are closely related and connected in most projects and thus are not clearly separable. User participation in the development and customization phase did take place when developers used agile methods. For instance, users participated by providing feedback on prototypes in intermittent user feedback cycles. Furthermore, we assessed the participation in the implementation and testing phase. This was user participation during training and adjustment efforts. We also judge if the user group participating in the project was representative for all users of the implemented IS. We distinguished between groups that were ‘not representative’ and ‘largely non-representative’, meaning that for instance several departments were affected by the new software, but only a small number of users of one department participated, and groups being representative for the user group, meaning that a higher number of users of affected departments was included. ‘Representative, few users’ was assigned if the participating users indeed represented the whole user group from a functional point of view but were only a very small share in respect of the whole user group.

We also assessed the user involvement in the different projects based on users’ perception of attributes of the IS. For instance, we evaluated whether users saw the software as a burden or as important and personally relevant (see Table 2). The assessment of the representativeness of the users was based on the analysis of interviews for each case. As a second calibration method, we used transformational assignment. In this approach, we made use of continuous functions to map base variable values to fuzzy values. Thus, we had to define three thresholds, one for full exclusion, the crossover threshold, and one for full inclusion [42]. While the full exclusion value defines the threshold of a condition for not being a member of a set (=0), the full inclusion value defines the threshold for a full member (=1).

Table 2. Conditions and outcome for the fsQCA.
Full size table

Moreover, the crossover threshold defines the boundary between a condition being a set member and not being a set member. We based our transformational assignment on a positive end-point concept, which implies that the set membership scores increase with increasing values of the base variable. We used transformational assignment for the calibration of the outcome of usability from the survey data in our study based on the calculation in R with the QCA package [42]. Specifically, we used the piecewise logistic function, which is the standard function in the QCA package in R. We set the full exclusion threshold to 50, the crossover threshold to 62, and the full inclusion threshold to 73 [45]. The raw data matrix containing the assessed conditions and outcome for all projects (Table 2) shows the results of the within-case analysis, whereas Table 3 contains all fuzzy-values on which we base the fsQCA.

Table 3. Overview of data for truth table minimization.
Full size table

4 Findings

When conducting a cross-case analysis using fsQCA, it is the primary goal to identify necessary and sufficient conditions for the examined outcome [7]. A condition is necessary if it is present in all cases in which the outcome is present. A consistency value of 1 [7], or at least higher than 0.9 [46] can indicate a necessary condition. We did not identify a necessary condition in this step of the analysis (see Table 4).

Table 4. Truth table for high system usability.
Full size table

This test is essential, since a condition which is necessary for an outcome as well as for the negated outcome is a trivial condition [7]. We also conducted an analysis for the sufficient conditions. A condition is sufficient if it is part of the configuration of the considered outcome in any case, implying that there is no case among the considered ones where the condition is present, but not the outcome. The sufficient conditions for the outcome are identified by the creation of a truth table (see Table 4) followed by using the enhanced Quine-McCluskey algorithm to minimize the Boolean output function [47]. We also conducted this analysis for the negative outcome to make sure that there are not contradictory paths in comparison to the positive outcome [7]. The required levels of consistency and subsequently coverage determine the inclusion of a configuration. We used a minimum consistency value of 0.8 for analyzing sufficiency, which is above the suggested minimal threshold of 0.75 [7, 47].

The application of the Quine-McCluskey algorithm in the aforementioned software solutions generates complex, parsimonious and intermediate solutions (see Table 5). We focus on the intermediate solution for our analysis because of the theoretical underpinnings of the minimization process [47]. In our particular case, the results for the intermediate and the parsimonious solution are identical, which indicates that logical remainders did not have a particular influence on the results of the minimization process. We focus our description of the analysis on the identical parsimonious and intermediate solution UPR*DUR. They suggest that only the combination of the conditions of early user participation during the requirements phase conducted with the appropriate users is relevant for project success in terms of usability. Besides its high level of consistency, this solution covers many different cases and therefore provides a good explanation for a large share of the outcome. Furthermore, this result indicates that user participation as a condition of IS project success is inseparable from the assessment of the appropriate users for the participatory practice. The projects with a membership in this solution all have some form of consultative or participative user participation in the requirements analysis phase.

Table 5. Truth table minimization results.
Full size table

The results of the fsQCA underline that UIP is more effective when the participants represent a large share of the affected users, a finding that links well to previous research results [30]. In sum, project success is very likely for the majority of the cases when the appropriate users participate, and this takes place in the phase of requirements analysis. This is our key finding which reinforces findings based on anecdotal evidence that claimed that user participation of representative end-users is related to IS implementation projects success. Thus, the research in this paper has two major contributions. First, it provides a richer perspective on UIP configurations related with IS project success. Second, it reaffirms prior research results in a more comprehensive way with an analysis of the configurations of different forms and timing of UIP in relation to IS implementation project success. In addition, our focus on perceived usability for end-users as a measure for success is a new approach that veers of the traditional time, budget, and quality approach towards a more end-users focused approach in the evaluation of IS implementation project success.

5 Discussion

Our analysis of multiple cases provides a detailed look at comprehensive patterns that link different forms of UIP across different project phases to IS project success. Through our novel configurational perspective, we contribute to research with an improvement in the level of detail of the understanding and empirical backing for the notion that user participation and involvement is especially beneficial in the requirements analysis phase [8]. Thus, our fsQCA provides additional empirical evidence on this finding. Furthermore, a higher level of user participation without participation of the appropriate user group (mainly end users) is unlikely to be associated with a significant improvement in the outcome. This finding can also be linked to previous research, which indicated that ineffective management of the participation and involving possibly the wrong users could also have adverse effects on project success [8]. This indicates the need to involve actual end-users and not just their managers or respective power users, who are not representative for the majority of the user group. The results of our analysis also cast doubt on the net effect of user participation in development as well as testing. However, a qualification of this assessment is necessary, as the lack of participation in the development phase can be due to the lack of prominence of truly agile development/customization approaches in most projects. Nonetheless, our results reinforce the notion that the participation of users in the development phase is much more prone to complications, therefore costly, and less effective than user participation in the requirements analysis phase. The low coverage of the complex solution, which encompasses user participation during development, indicates this. We also add to the research domain by providing an analysis of the intertwined conditions of user participation in the requirements phase and the appropriate degree of user representation. Our research also improves the understanding when and under which conditions well-established heuristics actually help to explain or even predict project success. It can be the objective of future research to evolve the theoretical understanding of the observed relationships.

However, there are also limitations for the interpretation of the results of our multiple case study. The number of cases that we were able to analyze, and their particular nature influenced the analysis. Furthermore, we used a limited number of conditions for the analysis. Only a larger number of cases can help to reduce these limitations. A larger number of cases would allow introducing more conditions in the QCA, or a different focus of a reanalysis of the available data set with a different theoretical motivation. A different theoretical perspective could, as aforementioned, include the condition of project complexity as it has already been established that the degree of user participation and involvement can be related to the complexity of the project [20]. Furthermore, we need to address the perception that there are possible threats to validity for the empirical analysis presented in this paper. First, the fsQCA is a deterministic technique for logical analysis, which means that the analysis results show that a particular condition, in our case user participation in the requirements analysis phase, most commonly occurs together with the outcome of a high level of usability as perceived by end-users. The observation of this particular configuration as parsimonious solution for the minimization process of the truth table does not mean that the user participation in the requirements analysis phase caused the high level of perceived usability. This can only be deducted from the observation based on the qualitative information from the cases and prior anecdotal evidence that suggests such an interpretation. Thus, there is no direct causal link, but the observation that a particular condition is almost always part of the configuration with a high level of usability. Second, the variation in the level of perceived usability by end-users cannot only be attributed to the conditions that we covered in the analysis presented in the paper, but also other factors. These other factors might include as aforementioned the project complexity, the specificity of the software solution and the user group. However, as fsQCA is designed to determine logical relationships it is not a probabilistic approach that causally determines an influence on a relationship. Thus, there is no direct threat to the validity of our results, as the results are only an indication based on logical analysis of empirical data.

Our analysis of the cases also has several practical implications. Project managers should make sure that they focus their attention on user participation in the phase of user requirements analysis. In particular, they should make sure that actual end-users of the IS participate. This should help to increase the perceived usability of the software and thereby increase the productivity of end users.

6 Discussion

We conducted a multiple-case study and highlighted the effect of UIP in the requirements phase on IS project success. We are able to show that the participation of the appropriate users in the requirements analysis phase is a key condition for IS project success. In addition, we are able to show that a higher level of user participation will not result in a significant improvement in the outcome, if users participate that cannot contribute as much as true end-users can. While these findings are a first empirical basis for the analysis of the relationship of different forms of end-user participation in IS implementation projects, further research on more different cases is still needed to broaden the theoretical basis for a preference for user participation in the requirements analysis phases. Specifically, the different forms of user participation in this particular project phase and their relationship with project success as experienced by end-users can be the subject of future research.