Evaluating the service quality of software providers appraised in CMM/CMMI

Abstract

Recently, several companies have decided to adopt maturity models such as the CMM/CMMI to ensure quality software processes. The state year report of the Software Engineering Institute (SEI) showed that more than three thousand CMMI appraisals have been conducted since 2002. Many of these were performed at software providers, i.e. companies that develop software for other companies. Although the costs of the implementation and appraisal of CMM/CMMI are high for the software providers, there is no formal study investigating whether this investment pays off or, in other words, whether their customers are measurably satisfied with the quality of the service provided. This article presents the results of a formal evaluation of customer perception of the service quality offered by the software providers appraised in CMM/CMMI. We developed an instrument based on a widely used service quality evaluation model (SERVQUAL) and applied this instrument to several customers of software providers appraised in CMM/CMMI. The results show a considerable discrepancy between customers’ expectations and their perceptions of the services provided.

Appendix A—customizing the instrument to our purpose

To assess the perceived service quality of the software providers appraised in CMMI/CMMI, three premises were set: (i) to use the SERVQUAL and IS-SERVQUAL instruments as references, taking into account that SERVQUAL is a generic model extensively used in the service quality area, and that the IS-SERVQUAL is a customization of the generic model for the information systems area, which is the closest to the software development area; (ii) to use CMMI SE/SW 1.1 as a reference, since CMM was discontinued; and (iii) to define a limited number of items to simplify the assessment.

In order to deal with the first premise, we used the items from both models (SERVQUAL and IS-SERVQUAL) as references to define the items for our instrument. However, the “tangibles” dimension was excluded—as for the IS-SERVQUAL instrument—in light of the fact that companies are increasingly adopting an off-shore model of development involving a kind of software factory where in many cases customers do not have contact with the providers’ installations.

In order to meet the second premise, we performed a detailed analysis in the CMMI’s process areas of specific goals and practices, seeking some correlation with each item from both models. This analysis was done in three steps. First, we analyzed the CMMI areas with the SERVQUAL dimensions of service quality. Then, we correlated the practices of each CMMI area with those dimensions. Finally, we refined the analysis to correlate each specific practice with each item of the models. This analysis was performed with the goal of answering the question: “if we institute this practice in my company, what level of quality should my customer expect or perceive of our service?” and “what is the influence of this practice on service quality?” For example, we determined that the requirements management area (REQM) is correlated with the reliability and assurance dimensions—the first, because with REQM the requirements can be controlled and service can thus be performed accurately; second, because with REQM it is possible to identify inconsistencies between requirements and work products, and it can thus inspire confidence and trust in the customer. The third step of the analysis was the most detailed, because we looked for some correlation between each practice and the service quality models items. In so doing, software practices that influence some items defined in the SERVQUAL model to assess the quality of service were identified. An example of one of these correlations is the REQM specific practice 1.1, “Obtain an Understanding of Requirements.” This practice was correlated to the “reliability” dimension item that says providers’ employees should be helpful and make customers at ease while discussing their needs. It was also correlated to the “empathy” dimension item that deals with the understanding of customers’ needs by the provider’s employees. All the practices from CMMI levels 2 and 3 were analyzed, since we considered them to be the bases of software engineering. As a result of this process, we obtained 281 correlations. Each of them is concerned, therefore, with one or more CMMI software practices, ensuring that the instrument will be evaluating software activities.

The instrument was defined with all its correlations, customizing items from the original models to represent what we would like to evaluate about the service quality of a provider that has CMM/CMMI. This customization involved expressing that the focus was about the service provider and software engineering aspects defined in CMMI. Sometimes, to clarify, examples of activities or components related to the software practice on items descriptions were included.

Finally, in order to limit the number of items (last premise), we reviewed all of the items; we made a grouping of the statements that had similar meanings and made them more general. They were consolidated into 50 items in total (25 for expectation and 25 for perception).

To verify the clarity of the items being evaluated, a semantic validation of the instrument was performed. Five specialists in CMMI from different institutions were invited to a brainstorming meeting: three were project leaders from CMMI level 5 providers; one had a PhD in software engineering and works with software quality; the last one was a customer of CMMI providers. They were selected for their profiles and their knowledge about CMMI from the list of professors and Master’s students in information technology from the Catholic University. At the beginning, each specialist read the instrument and wrote down his/her comments. Then, a moderator read each item aloud and the specialists commented and discussed whether the item clearly imparted the information needed regarding what was being assessed. As a result of this meeting, a list of improvement suggestions for item writing was achieved. Table 2 presents the final version of the instrument with the expectation statements. The perception statement exhibit the same content, but with a change in focus to reflect customers actually perceived regarding service quality after the service was rendered.