Abstract
In this paper, data envelopment analysis variable returns to scale (DEA VRS) model is applied to data collected on 79 software development projects from a leading CMMI level 5 organization. We divide overall software effort into software development effort, software quality conformance effort (EoC), and software maintenance non-conformance (EoNC) effort due to poor software quality at delivery time. Partitioning effort into software development and software quality metrics provides us a comprehensive model to measure productivity of software projects and to identify best practice projects. Some of positive productivity drivers from the DEA best practice efficient projects point to good customer rapport and application familiarity. Inefficient projects had problems such as customer requirements volatility, and the use of unfamiliar technology. The DEA results identify 12 “best practice” projects that can be emulated for software process improvement. Additionally, our results point to approximately 50 % potential for productivity improvement in software projects to get to the level of “best practice” projects. This study shows that including EoC and EoNC as inputs has a positive impact on the best practice frontier.
Similar content being viewed by others
References
Hill PR (2010) Practical software project estimation: a toolkit for estimating software development effort & duration. McGraw-Hill, San Francisco
Jalote P (2004) CMM in practice: processes for executing software projects at Infosys. Pearson, Delhi
Flitman A (2003) Towards meaningful benchmarking of software development team productivity. Benchmarking Int J 10(4):382–399
Paradi JC, Reese DN, Rosen D (1997) Applications of DEA to measure the efficiency of software production at two large Canadian banks. Ann Oper Res 73:91–115
Parthasarathy S, Anbazhagan N (2008) Evaluating ERP projects using DEA and regression analysis. Int J Bus Inf Syst 3(2):140–157
Stensrud E, Myrtveit I (2003) Identifying high performance ERP projects. IEEE Trans Comput 29(5):398–416
Yang Z, Paradi JC (2004) DEA evaluation of a Y2K software retrofit program. IEEE Trans Eng Manag 51(3):279–287
Card DN, El Emam K, Scalzo B (2001) Measurement of object-oriented software development projects. Software Productivity Consortium NFP, Herndon
Basili VR, Briand LC, Melo WL (1996) How reuse influences productivity in object-oriented systems. Commun ACM 39(10):104–116
Blackburn JD, Scudder GD, Van Wassenhove LN (1996) Improving speed and productivity of software development: a global survey of software developers. IEEE Trans Softw Eng 22(12):875–885
Briand LC, El-Emam K, Bomarius F COBRA (1998) A hybrid method for software cost estimation, benchmarking and risk assessment. In: 20th international conference on software engineering, pp 390–399
Jeffery R, Ruhe M, Wieczorek I (2001) Using public domain metrics to estimate software development effort. In: Proceedings of METRICS’01, pp 16–27
Boehm B (1981) Software engineering economics. Prentice-Hall, Englewood Cliffs
Boehm B, Clark B, Horowitz E, Westland C, Madachy R, Selby R (1996) The Cocomo 2.0 software cost estimation model—a status report. Am Program. 9:2–17
Banker RD, Kemerer CF (1989) Scale economies in new software development. IEEE Trans Softw Eng 15(10):1199–1205
Pendharkar PC, Rodger JA, Subramanian GH (2008) An empirical study of the Cobb–Douglas production function properties of software development effort. Inf Softw Technol 50:1181–1188
Kneuper R (2008) Improving software and systems development processes using capability maturity model integration (CMMI-DEV). Rocky Nook, Santa Barbara
Goldenson DR, Gibson DL (2003) Demonstrating the impact and benefits of CMMI: an update and preliminary results. CMU/SEI-2003-SR-009. Carnegie Mellon Software Engineering Institute, Pittsburgh
Charnes A, Cooper WW, Lewin AY, Seiford LM (1994) Data envelopment analysis: theory, methodology, and applications. Kluwer, Boston
Charnes A, Cooper WW, Golany B, Seiford L, Stutz J (1985) Foundations of data envelopment analysis for Pareto-Koopmans efficient empirical production functions. J Econom 30:91–107
Emrouznedjad A, Parker BR, Tavares G (2008) Evaluation of research in efficiency and productivity: a survey and analysis of the first 30 years of scholarly literature in DEA. Soc Econom Plan Sci 44:151–157
Seiford LM (1996) Data envelopment analysis: the evaluation of the state of the art (1978–1995). J Prod Anal 9:99–137
Seiford LM, Thrall RM (1990) Recent developments in DEA: the mathematical programming approach to frontier analysis. J Econom 46:7–38
Tavares G (2002) A bibliography of data envelopment analysis (1978–2001). RUTCOR, Rutgers University, Newark
Charnes A, Cooper WW, Rhodes E (1978) Measuring the efficiency of decision making units. Eur J Oper Res 2:429–444
Banker RD, Charnes A, Cooper WW (1984) Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manag Sci 30(9):1078–1092
Banker RD, Datar SM, Kemerer CF (1991) A model to evaluate variables impacting the productivity of software maintenance projects. Manag Sci 37(1):1–18
Elam J (1991) Evaluating the efficiency of IS organizations using data envelope analysis. In: Proceedings of the International Function Point Users Group
Banker RD, Chang H, Kemerer CF (1994) Evidence on economies of scale in software development. Inf Softw Technol 36(5):275–282
Mahmood MA, Pettingell KJ, Shaskevich AI (1996) Measuring productivity of software projects: a data envelope analysis approach. Decis Sci 27(1):57–80
Myrtveit I, Stensrud E (1999) Benchmarking COTS projects using data envelopment analysis. In: Proceedings of METRICS’99, Washington, D.C., pp 269–278
Zhang S, Wang Y, Tong J, Zhou J, Ruan L (2006) Evaluation of project quality: a DEA-based approach. Lect Notes Comput Sci 3966:88–96
Asmild M, Paradi JC, Kulkarni A (2006) Using data envelopment analysis in software development productivity measurement. Softw Process Improv Pract 11:561–572
Wray B, Mathieu R (2008) Evaluating the performance of open source software projects using data envelopment analysis. Inf Manag Comput Secur 16(5):449–462
Maxwell K, Forselius P (2000) Benchmarking software development productivity. Softw IEEE 17(1):80–88
Pendharkar PC, Rodger J (2009) The Relationship between software development team size and software development cost. Commun ACM 52(1):141–144
McCall JA, Richards PK, Walters GF (1977) Factors in software quality. Volumes 1, 2, 3, US Rome Air Development Center Reports NTIS AD/A-049 014, 015, 055
Boehm BW, Brown JR, Kaspar JR, Lipow M, MacCleod GJ, Merrit MJ (1978) Characteristics of software quality. North-Holland, Amsterdam
Fenton NE (1994) Software measurement: a necessary scientific basis. IEEE Trans Softw Eng 20(3):199–206
Kemayel L, Mili A, Ouederni I (1991) Controllable factors for programmer productivity: a statistical study. J Syst Softw 16(2):151–163
Zhu J (2003) Quantitative models for performance evaluation and benchmarking: data envelopment analysis with spreadsheets. Kluwer, Boston
Bowlin WF (1998) Measuring performance: an introduction to data envelopment analysis (DEA). J Cost Anal 7:3–27
Chen Y, Zhu J (2004) Measuring information technology’s indirect impact on firm performance. Inf Technol Manag 5(1):9–22
Soteriou AC, Zenios SA (1998) Data envelopment analysis: an introduction and an application to bank branch performance assessment. In: Marcoulides G (ed) Modern methods for business research. Lawrence Erlbaum Associates, London
Drake L, Howcroft B (1994) Relative efficiency in the branch network of a UK bank: an empirical study. Omega 22(1):83–90
Cooper WW, Seiford LM, Zhu J (2004) Handbook on data envelopment analysis. Springer, New York
Sexton TR, Silkman RH, Hogan AJ (eds) (1986) Data envelopment analysis: Critique and extensions, new directions for program evaluation. Jossey-Bass, San Francisco
Andersen P, Petersen NC (1993) A procedure for ranking efficient units in data envelopment analysis. Manag Sci 39:1261–1264
Seiford LM, Zhu J (1999) Infeasibility of super efficiency data envelopment analysis models. INFOR 37(2):174–187
Torgersen AM, Førsund FR, Kittelsen SA (1996) Slack adjusted efficiency measures and ranking of efficient units. J Prod Anal 7:379–398
Boh WF, Slaughter SA, Espinosa JA (2007) Learning from experience in software development: a multilevel analysis. Manag Sci 53(8):1315–1331
Keil M, Cule PE, Lyytinen K, Schmidt RC (1998) A framework for identifying software project risks. Commun ACM 41(11):76–83
Mantle MW, Lichty R (2012) Managing the unmanageable: rules, tools, and insights for managing software people and teams. Addison-Wesley, Boston
Kittelsen S (1999) Monte Carlo simulations of DEA efficiency measures and hypothesis tests. Department of Economics, University of Oslo, Oslo
Pendharkar PC, Rodger JA (2007) An empirical study of the impact of team size on software development effort. Inf Technol Manag 8:253–262
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Pai, D.R., Subramanian, G.H. & Pendharkar, P.C. Benchmarking software development productivity of CMMI level 5 projects. Inf Technol Manag 16, 235–251 (2015). https://doi.org/10.1007/s10799-015-0234-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10799-015-0234-4