Actual vs. perceived effect of software engineering practices in the Italian industry

doi:10.1016/j.jss.2010.05.077

Journal of Systems and Software

Volume 83, Issue 10, October 2010, Pages 1907-1916

https://doi.org/10.1016/j.jss.2010.05.077 Get rights and content

Abstract

A commonly cited limitation of software engineering research consists in its detachment from the industrial practice. Several studies have analyzed a number of practices and identified their benefits and drawbacks but little is known about their dissemination in the industry. For a set of 18 practices commonly studied in the literature, this paper investigated diffusion, effect on the success, and perceived usefulness in 62 actual industrial projects from 28 Italian IT companies. In particular we proposed a classification of these perceptions and we were able to classify 14 practices. We found statistical evidence that 7 factors have an actual effect (positive for 6 of them, negative for one). Moreover 77% (10 out of 13) of the known good practices (e.g., importance of good project schedule or complete requirements’ list) are perceived consistently by the industry. For a few other practices (having a champion's support, using metrics, reducing quality) we noticed a lack of awareness in the industry. Starting from these observations we propose guidelines for industrial practice and suggestions for academic research.

Introduction

According to the IEEE glossary “Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software”. Usually it encompasses the activities of a team led by a project manager and the interaction with customers in a competitive and volatile environment.

The first problems with software development were identified during the NATO Garmisch conference in 1968. Since then academic research is focused on defining better practices for software development. However, the rate of software project failures remains high (CHAOS).

Researchers identified several software engineering practices – both useful and harmful – and their expected effects on a software process outcome. Unfortunately the actual adoption of practices is not driven by scientific empirical knowledge. One reason for this could be poor communication between academic and industrial worlds. Single case studies and joint projects do not provide complete feedback from practitioners on the adopted practices and their usefulness or applicability to daily software development process. At the same time academy keeps developing solutions that perform well in some projects, but there is little knowledge about how real or utopian these solutions are in industrial software development.

Among the factors influencing the adoption of software engineering practices an important role is played by their perceived usefulness. This is one of the main factors identified in the technology acceptance model (TAM). TAM provides a framework describing how users come to accept and use a technology (Davis, 1989).

The main goal of the work presented in this paper is to assess the perceived effects of several well-known software engineering practices on the success or failure of the projects. More in detail we focused on answering the following two research questions:

RQ1: Which factors do affect the success or failure of software projects?
RQ2: What is the perceived effect of presence (or absence) of a factor X on a project's outcome?

The assessment was performed on the basis of a series of interviews with practitioners working in the industry. In particular, as far as the practitioners’ perception is concerned (RQ2), we devised a classification schema in order to objectively evaluate the appreciation of best practices.

The paper provides several contributions:

•
it presents additional empirical evidence on the objective effectiveness of best practices,
•
it describes the industrial acknowledgment of best practices, and
•
it defines a taxonomy for the perception of practices.

The paper is organized as follows: Section 2 presents the background of the study, the conceptual model and gives an overview of related work. In Section 3 we define the study, including factor taxonomy, hypotheses formulation, sampling strategy, data measurement and analysis techniques. Section 4 presents data analysis. In Section 5 we discuss the findings of the study. In Section 6 we address threats to validity. Conclusion is presented in Section 7.

Section snippets

Background and related work

In this section we describe studies related to the analysis of software engineering practices and their industry-wide adoption. According to the PMBOK Guide (PMI, 2004) “A project is a temporary endeavor undertaken to create a unique product, service, or result”. We use this definition for software projects too. We organize the observable characteristics of a project into a conceptual model, presented in Fig. 1. While we adopt – on purpose – a terminology that is consistent with that used in

Study definition

In this section we present the detailed research hypotheses. Before that we present a model to evaluate the effect of factors that will be used in defining the hypotheses.

Descriptive statistics

We have gathered answers from 33 respondents, both project managers and developers, via phone and email contacts. In total we have data about 33 successful projects and 29 failed ones from 28 companies.³ Response rate was about 25%, which is good compared to similar studies. The selection of companies was performed through random sampling; we checked whether the distribution of company size in our sample resulted

Discussion

We have conducted a case–control study to verify the actual effectiveness of project success factors (RQ1) and the perception of the same factors by software developers (RQ2). Results are summarized in Table 8. Answers to RQ1 are either “Yes” or “No”, indicating whether the factor had a statistically significant influence on the outcome. Odds ratios are also reported: values greater than 1 indicate a success factor while values lower than 1 indicate a failure factor. Answers to RQ2 are the

Threats to validity

We discuss here the threats to validity of this study.

Conclusion validity concerns the relationship between the treatment and the outcome. We have used proper non-parametric statistical tests for data analysis and, when necessary, applied α-level corrections to mitigate the possibility of Type I errors.

An important threat to internal validity is represented by the possible lack of accuracy in independent variables’ measurement. The independent variables consisted in determining whether a given

Conclusions

This paper reported results from a case–control study aimed at understanding the effect of some well-known practices on the outcome of software projects. We have analyzed and compared both the actual effect and the perceived effect of these factors. For a number of factors there is complete agreement between perceived and actual effect, and the effect is statistically significant. We conclude that these practices should be recommended for all software projects: defining precisely the

Evgenia Egorova is a Post-Doc researcher at the Politecnico di Torino, Italy. Her research interests include empirical software engineering, software metrics, and software product and project management. She received PhD degree from the Politecnico di Torino, MSc degree from the Technical University Hamburg-Harburg, Germany, and Bachelor's degree from the Moscow State University of Economics, Statistics and Informatics, Russia. She spent one year working at Siemens Mobiles, Munich, Germany.

References (36)

K.R. Linberg
Software developer perceptions about software project failure: a case study
Journal of Systems and Software
(1999)
A. Agresti
An Introduction to Categorical Data Analysis
(2007)
R. Berntsson-Svensson et al.
Successful software project and products: an empirical investigation
Proceedings of the ACM/IEEE International Symposium on Empirical Software Engineering
(2006)
V. Basili et al.
The future engineering of software: a management perspective
IEEE Computer Magazine
(1991)
F. Brooks
The Mythical Man-Month: Essays on Software Engineering
(1995)
Camera di commercio (CCIAA) di Torino – UNIMATICA di Torino, 2006. L’ICT in Provincia di Torino: La sfida...
M. Cantamessa et al.
Il settore ICT in Piemonte
(2004)
D. Coggon et al.
Epidemiology for the Uninitiated
(1997)
CHAOS reports from the Standish Group http://www.standishgroup.com. Biennial reports require subscription, sample from...
D. Damian et al.
An industrial case study of immediate benefits of requirements engineering process improvement at the Australian Center for Unisys Software
Empirical Software Engineering
(2004)

F.D. Davis

Perceived usefulness, perceived ease of use, and user acceptance of information technology

MIS Quarterly

(1989)

DeMarco, T., Lister, T., 1987. Peopleware: Productive Projects and Teams, Dorset...

T. Dybå

Factors of software process improvement success in small and large organizations: an empirical study in the Scandinavian Context

T. Field

When BAD things happen to GOOD projects

CIO Magazine

(1997)

R.A. Fisher

Statistical Methods for Research Workers

(1954)

C. Holland et al.

A critical success factors model for ERP implementation

IEEE Software

(1999)

L. Huang et al.

Determining how much software assurance is enough: a value-based approach

Proceedings of the 7th International Workshop on Economics-driven Software Engineering Research

(2005)

F. Hubert et al.

Requirements engineering as a success factor in software projects

IEEE Software

(2001)

Cited by (13)

Evaluating different families of prediction methods for estimating software project outcomes
2016, Journal of Systems and Software
Citation Excerpt :
Most previous research on the prediction of software project outcome has focused on the identification of the software development characteristics or practices which contribute to project success (Abe et al., 2006; Cerpa et al., 2010) or in the probability of occurrence of threats (Smite, 2007). There are characteristics or practices that should be used by all software projects in order to be successful, but others may have different effects depending on the context of the project (Egorova et al., 2010). Some researchers suggest that it is essential to identify the characteristics of successful projects in order to determine similarities (Cerpa et al., 2010; Procaccino et al., 2001; Wohlin and Andrews, 2005).
Software has been developed since the 1960s but the success rate of development projects is still low. Classification models have been used to predict defects and effort estimation, but little work has been done to predict the outcome of these projects. Previous research shows that it is possible to predict outcome using classifiers based on key variables during development, but it is not clear which techniques provide more accurate predictions. We benchmark classifiers from different families to determine the outcome of a software project and identify variables that influence it. A survey-based empirical investigation was used to examine variables contributing to project outcome. Classification models were built and tested to identify the best classifiers for this data by comparing their AUC values. We reduce the dimensionality of the data with Information Gain and build models with the same techniques. We use Information Gain and classification techniques to identify key attributes and their relative importance. We find that four classification techniques provide good results for survey data, regardless of dimensionality reduction. We conclude that Random Forest is the most appropriate technique for predicting project outcome. We identified key attributes which are related to communication, estimation, and process review.
Cross-factor analysis of software engineering practices versus practitioner demographics: An exploratory study in Turkey
2016, Journal of Systems and Software
Citation Excerpt :
For example, a recent paper (Garousi and Zhi, 2013) gathered the opinions of 246 Canadian software engineers in 2013 about software testing practices and reported various findings, e.g., (1) The importance of testing-related training is increasing over years, (2) Functional and unit testing are two common test types that receive the most attention and effort. While the data and findings reported by existing surveys, (e.g., Beck and Perkins, 1983, Zelkowitz et al., 1984, Curtis et al., 1988, Cusumano and Kemerer, 1990, Blackburn et al., 1996, Singer et al., 1997, Holt, 1997, Dutta et al., 1999, Groves et al., 2000, Cusumano et al., 2003, Aytaç et al., 2003, Denger et al., 2007, Aykol, 2009, Sökmen, 2010, Egorova et al., 2010, Kirk and Tempero, 2012, Vonken et al., 2012, Garousi et al., 2014, Garousi and Zhi, 2013), are interesting, most of those studies have studied SE practices individually, e.g., software testing practices in Garousi and Zhi (2013). In addition to such trends and findings, it is also important to understand the cross-factor relationship of SE practices, e.g., what are the correlations among the degree of challenges experienced in various SE phases/tasks?
Understanding the types of software engineering practices and techniques used in the industry is important. There is a wide spectrum in terms of the types and maturity of software engineering practices conducted in each software team and company. Furthermore, it is important to understand the cross-factor relationship of software engineering practices and practitioner demographics including their companies and projects, e.g., is it the case that practitioners developing software for sectors such as military would utilize software size measurement approaches more, or use different software development methodologies, compared to practitioners developing software for other sectors?, and what kinds of practices are conducted by practitioners in small versus large companies?
Our objective is to get an understanding into the cross-factor correlation of various software engineering practices versus practitioner demographics including their companies and projects (e.g., target industry, size and work experience). Such an understanding will enable us to identify patterns and pinpoint special issues which should be studied and addressed in the context of each specific demographic (e.g., small versus large companies). Thus, we decided to conduct an exploratory study in this area and collected real industrial data in the context of Turkey which has a vibrant software industry.
To achieve the above objective, we use the data from a recent Turkish-wide survey of software engineering practices which was systematically designed with 46 questions based on our past experience in the Canadian context and using the Software Engineering Body of Knowledge (SWEBOK). 202 practicing software engineers participated in the survey. We raise a set of 12 research questions about the cross-factor correlation of software engineering practices and practitioner demographics, and address them using statistical analysis.
The exploratory study results reveal important and interesting findings about cross-factor relationship of software engineering practices and practitioner demographics. Among some of the most interesting findings are the followings: (1) By analyzing the trends, we were first surprised to see that as a practitioner gets more years of work experience, against what one would expect, s/he experiences more challenges in Software Development Life-Cycle (SDLC) phases and SE tasks; (2) Almost 55% of participants measure software size; (3) Agile/lean development is used the least (16%) by the participants working in the companies serving the military and defense sector; (4) Usage of waterfall is low among participants employed by small-sized companies whereas Agile/lean development is relatively popular among this class of participants; and (5) As company size increases, usage of spiral development slightly increases, whereas usage of extreme programming practices decreases.
The results of this exploratory study will be useful to software engineering professionals and researchers both in Turkey and world-wide by revealing the cross-factor relationship of software engineering practices versus practitioner demographics. The study raises several new research directions, e.g., (1) Why are not many practitioners using any size nor project estimation metrics and how these fundamental engineering approaches could be utilized more frequently?, (2) What are the best practices, success stories and challenging experiences in using SE tools?, and (3) Why is Agile/lean development is not popular in the military and defense sector and how these approaches could be utilized more frequently.
A survey of software engineering practices in Turkey
2015, Journal of Systems and Software
Understanding the types of software engineering (SE) practices and techniques used in industry is important. There is a wide spectrum in terms of the types and maturity of SE practices conducted in industry. Turkey has a vibrant software industry and it is important to characterize and understand the state of its SE practices. Our objective is to characterize and grasp a high-level view on type of SE practices in the Turkish software industry. To achieve this objective, we systematically designed an online survey with 46 questions based on our past experience in the Canadian and Turkish contexts and using the Software Engineering Body of Knowledge (SWEBOK). Two hundred and two practicing software engineers from the Turkish software industry participated in the survey. The survey results reveal important and interesting findings about SE practices in Turkey and beyond. They also help track the profession of SE, and suggest areas for improved training, education and research. Among the findings are the followings: (1) The military and defense software sectors are quite prominent in Turkey, especially in the capital Ankara region, and many SE practitioners work for those companies. (2) 54% of the participants reported not using any software size measurement methods, while 33% mentioned that they have measured lines of code (LOC). (3) In terms of effort, after the development phase (on average, 31% of overall project effort), software testing, requirements, design and maintenance phases come next and have similar average values (14%, 12%, 12% and 11% respectively). (4) Respondents experience the most challenge in the requirements phase. (5) Waterfall, as a rather old but still widely used lifecycle model, is the model that more than half of the respondents (53%) use. The next most preferred lifecycle models are incremental and Agile/lean development models with usage rates of 38% and 34%, respectively. (6) The Waterfall and Agile methodologies have slight negative correlations, denoting that if one is used in a company, the other will less likely to be used. The results of our survey will be of interest to SE professionals both in Turkey and world-wide. It will also benefit researchers in observing the latest trends in SE industry identifying the areas of strength and weakness, which would then hopefully encourage further industry–academia collaborations in those areas.
Failure factors of small software projects at a global outsourcing marketplace
2014, Journal of Systems and Software
The presented study aims at a better understanding of when and why small-scale software projects at a global outsourcing marketplace fail. The analysis is based on a data set of 785,325 projects/tasks completed at vWorker.com. A binary logistic regression model relying solely on information known at the time of a project's start-up correctly predicted 74% of the project failures and 67% of the non-failures. The model-predicted failure probability corresponded well with the actual frequencies of failures for most levels of failure risk. The model suggests that the factors connected to the strongest reduction in the risk of failure are related to previous collaboration between the client and the provider and a low failure rate of previous projects completed by the provider. We found the characteristics of the client to be almost as important as those of the provider in explaining project failures and that the risk of project failure increased with an increased client emphasis on low price and with an increased project size. The identified relationships seem to be reasonable stable across the studied project size categories.
Perceived causes of software project failures - An analysis of their relationships
2014, Information and Software Technology
Citation Excerpt :
In the case of software project failure, the project environment is commonly related to the project complexity [18,42,43], organizational factors [37], available assets [12,37,42–44], policies [43], structure [43], business domain [27,37,45] and technology [45]. The people related causes [27] cover social interaction [27,42,45,46], skills [13,42,45,46], and motivation [16,47]. McLeod and MacDonell [27] indicate that social interaction affects the outcome of software projects.
Software project failures are common. Even though the reasons for failures have been widely studied, the analysis of their causal relationships is lacking. This creates an illusion that the causes of project failures are unrelated.
The aim of this study is to conduct in-depth analysis of software project failures in four software product companies in order to understand the causes of failures and their relationships. For each failure, we want to understand which causes, so called bridge causes, interconnect different process areas, and which causes were perceived as the most promising targets for process improvement.
The causes of failures were detected by conducting root cause analysis. For each cause, we classified its type, process area, and interconnectedness to other causes. We quantitatively analyzed which type, process area, and interconnectedness categories (bridge, local) were common among the causes selected as the most feasible targets for process improvement activities. Finally, we qualitatively analyzed the bridge causes in order to find common denominators for the causal relationships interconnecting the process areas.
For each failure, our method identified causal relationships diagrams including 130–185 causes each. All four cases were unique, albeit some similarities occurred. On average, 50% of the causes were bridge causes. Lack of cooperation, weak task backlog, and lack of software testing resources were common bridge causes. Bridge causes, and causes related to tasks, people, and methods were common among the causes perceived as the most feasible targets for process improvement. The causes related to the project environment were frequent, but seldom perceived as feasible targets for process improvement.
Prevention of a software project failure requires a case-specific analysis and controlling causes outside the process area where the failure surfaces. This calls for collaboration between the individuals and managers responsible for different process areas.
Relevance, benefits, and problems of software modelling and model driven techniques - A survey in the Italian industry
2013, Journal of Systems and Software
Claimed benefits of software modelling and model driven techniques are improvements in productivity, portability, maintainability and interoperability. However, little effort has been devoted at collecting evidence to evaluate their actual relevance, benefits and usage complications.
The main goals of this paper are: (1) assess the diffusion and relevance of software modelling and MD techniques in the Italian industry, (2) understand the expected and achieved benefits, and (3) identify which problems limit/prevent their diffusion.
We conducted an exploratory personal opinion survey with a sample of 155 Italian software professionals by means of a Web-based questionnaire on-line from February to April 2011.
Software modelling and MD techniques are very relevant in the Italian industry. The adoption of simple modelling brings common benefits (better design support, documentation improvement, better maintenance, and higher software quality), while MD techniques make it easier to achieve: improved standardization, higher productivity, and platform independence. We identified problems, some hindering adoption (too much effort required and limited usefulness) others preventing it (lack of competencies and supporting tools).
The relevance represents an important objective motivation for researchers in this area. The relationship between techniques and attainable benefits represents an instrument for practitioners planning the adoption of such techniques. In addition the findings may provide hints for companies and universities.

View all citing articles on Scopus

Maurizio Morisio is an associate professor at the Politecnico di Torino, Turin, Italy. He received the PhD degree in software engineering and the MSc degree in electronic engineering from the Politecnico di Torino. He spent two years working with the Experimental Software Engineering Group at the University of Maryland, College Park. His current research interests include experimental software engineering, service engineering, software reuse metrics and models, agile methodologies, and commercial off-the-shelf processes and integration. He is a consultant for improving software production through technology and processes. He is a member of the IEEE Computer Society.

Marco Torchiano is an assistant professor at Politecnico di Torino, Italy; he has been post-doctoral research fellow at Norwegian University of Science and Technology (NTNU), Norway. He received an MSc and a PhD in Computer Engineering from the Politecnico di Torino, Italy. He published more than 50 research papers in international journals and conferences. He is the co-author of the book ‘Software Development—Case studies in Java’ from Addison-Wesley, and co-editor of the book ‘Developing Services for the Wireless Internet’ from Springer. His current research interests are: empirical software engineering, OTS-based development and software engineering for mobile and wireless applications.

View full text

Actual vs. perceived effect of software engineering practices in the Italian industry

Abstract

Introduction

Section snippets

Background and related work

Study definition

Descriptive statistics

Discussion

Threats to validity

Conclusions

Journal of Systems and Software

An Introduction to Categorical Data Analysis

Successful software project and products: an empirical investigation

Proceedings of the ACM/IEEE International Symposium on Empirical Software Engineering

The future engineering of software: a management perspective

IEEE Computer Magazine

The Mythical Man-Month: Essays on Software Engineering

Il settore ICT in Piemonte

Epidemiology for the Uninitiated

An industrial case study of immediate benefits of requirements engineering process improvement at the Australian Center for Unisys Software

Empirical Software Engineering

Perceived usefulness, perceived ease of use, and user acceptance of information technology

MIS Quarterly

Factors of software process improvement success in small and large organizations: an empirical study in the Scandinavian Context

When BAD things happen to GOOD projects

CIO Magazine

Statistical Methods for Research Workers

A critical success factors model for ERP implementation

IEEE Software

Determining how much software assurance is enough: a value-based approach

Proceedings of the 7th International Workshop on Economics-driven Software Engineering Research

Requirements engineering as a success factor in software projects

IEEE Software