Abstract
There exists a large and growing number of proposed estimation methods but little conclusive evidence ranking one method over another. Prior effort estimation studies suffered from “conclusion instability”, where the rankings offered to different methods were not stable across (a) different evaluation criteria; (b) different data sources; or (c) different random selections of that data. This paper reports a study of 158 effort estimation methods on data sets based on COCOMO features. Four “best” methods were detected that were consistently better than the “rest” of the other 154 methods. These rankings of “best” and “rest” methods were stable across (a) three different evaluation criteria applied to (b) multiple data sets from two different sources that were (c) divided into hundreds of randomly selected subsets using four different random seeds. Hence, while there exists no single universal “best” effort estimation method, there appears to exist a small number (four) of most useful methods. This result both complicates and simplifies effort estimation research. The complication is that any future effort estimation analysis should be preceded by a “selection study” that finds the best local estimator. However, the simplification is that such a study need not be labor intensive, at least for COCOMO style data sets.
Similar content being viewed by others
References
Anderson, J.: Cognitive Psychology and its Implications. W.H. Freeman, New York (1985)
Baker, D.: A hybrid approach to expert and model-based effort estimation. Master’s thesis, Lane Department of Computer Science and Electrical Engineering, West Virginia University, 2007. Available from https://eidr.wvu.edu/etd/documentdata.eTD?documentid=5443 (2007)
Basili, V., McGarry, F., Pajerski, R., Zelkowitz, M.: Lessons learned from 25 years of process improvement: the rise and fall of the NASA software engineering laboratory. In: Proceedings of the 24th International Conference on Software Engineering (ICSE) 2002, Orlando, Florida, 2002. Available from http://www.cs.umd.edu/projects/SoftEng/ESEG/papers/83.88.pdf
Boehm, B.: Software Engineering Economics. Prentice Hall, New York (1981)
Boehm, B.: A spiral model of software development and enhancement. Softw. Eng. Notes 11(4), 22 (1986)
Boehm, B., Horowitz, E., Madachy, R., Reifer, D., Clark, B.K., Steece, B., Brown, A.W., Chulani, S., Abts, C.: Software Cost Estimation with Cocomo II. Prentice Hall, New York (2000)
Briand, L., Langley, T., Wieczorek, I.: A replicated assessment and comparison of common software cost modeling techniques. In: Proceedings of the 22nd International Conference on Software Engineering, Limerick, Ireland, pp. 377–386 (2000)
Brieman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)
Chang, C.: Finding prototypes for nearest neighbor classifiers. IEEE Trans. Comput. C-23, 1179–1185 (1974)
Chen, Z., Menzies, T., Port, D., Boehm, B.: Finding the right data for software cost modeling. IEEE Softw. 22(6), 38–46 (2005)
Chulani, S., Boehm, B., Steece, B.: Bayesian analysis of empirical software engineering cost models. IEEE Trans. Softw. Eng. 25(4), 573–583 (1999)
Endres, A., Rombach, H.D.: A Handbook of Software and Systems Engineering—Empirical Observations, Laws and Theories. Addison-Wesley, Reading (2003)
Fayyad, U.M., Irani, I.H.: Multi-interval discretization of continuous-valued attributes for classification learning. In: Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, pp. 1022–1027 (1993)
Ferens, D., Christensen, D.: Calibrating software cost models to Department of Defense Database: a review of ten studies. J. Parametr. 18(1), 55–74 (1998)
Freund, Y., Schapire, R.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55 (1997)
Gama, J., Pinto, C.: Discretization from data streams: applications to histograms and data mining. In: SAC’06: Proceedings of the 2006 ACM Symposium on Applied Computing, pp. 662–667. ACM Press, New York (2006). Available from http://www.liacc.up.pt/~jgama/IWKDDS/Papers/p6.pdf
Gregor, S.: Design theory in information systems. Australasian J. Inf. Syst. (2002). Available from http://dl.acs.org.au/index.php/ajis/article/viewPDFInterstitial/439/399?ads=
Hall, M., Holmes, G.: Benchmarking attribute selection techniques for discrete class data mining. IEEE Trans. Knowl. Data Eng. 15(6), 1437–1447 (2003). Available from http://www.cs.waikato.ac.nz/~mhall/HallHolmesTKDE.pdf
Jalali, O.: Evaluation bias in effort estimation. Master’s thesis, Lane Department of Computer Science and Electrical Engineering, West Virginia University (2007)
Jensen, R.: An improved macrolevel software development resource estimation model. In: 5th ISPA Conference, pp. 88–92, April 1983
Jorgensen, M.: A review of studies on expert estimation of software development effort. J. Syst. Softw. 70(1–2), 37–60 (2004)
Jorgensen, M., Molokken-Ostvold, K.: Reasons for software effort estimation error: Impact of respondent error, information collection approach, and data analysis method. IEEE Trans. Softw. Eng. 30(12), 993–1007 (2004)
Jorgensen, M., Shepperd, M.: A systematic review of software development cost estimation studies. January 2007. Available from http://www.simula.no/departments/engineering/publications/Jorgensen.2005.12
Kemerer, C.: An empirical validation of software cost estimation models. Commun. ACM 30(5), 416–429 (1987)
Keung, J.W., Kitchenham, B.A., Jeffery, D.R.: Analogy-x: providing statistical inference to analogy-based software cost estimation. IEEE Trans. Softw. Eng. 34(4), 471–484 (2008)
Kirsopp, C., Shepperd, M.: Making inferences with small numbers of training sets. IEEE Proc. 149 (2002a)
Kirsopp, C., Shepperd, M.: Case and feature subset selection in case-based software project effort prediction. In: Proc. of 22nd SGAI International Conference on Knowledge-Based Systems and Applied Artificial Intelligence, Cambridge, UK (2002b)
Kitchenham, B.A., Mendes, E., Travassos, G.H.: Cross versus within-company cost estimation studies: a systematic review. IEEE Trans. Softw. Eng. 33(5), 316–329 (2007)
Kliijnen, J.: Sensitivity analysis and related analyses: a survey of statistical techniques. J. Stat. Comput. Simul. 57(1–4), 111–142 (1987)
Kohavi, R., John, G.H.: Wrappers for feature subset selection. Artif. Intell. 97(1–2), 273–324 (1997). [Online]. Available: citeseer.nj.nec.com/kohavi96wrappers.html
Li, J., Ruhe, G.: Decision support analysis for software effort estimation by analogy. In: Proceedings, PROMISE’07 Workshop on Repeatable Experiments in Software Engineering (2007)
Li, Y., Xie, M., Goh, T.N.: A study of the non-linear adjustment for analogy based software cost estimation. Empir. Softw. Eng., 603–643 (2009)
Li, Y., Xie, M., Goh, T.N.: A study of project selection and feature weighting for analogy based software cost estimation. J. Syst. Softw. 82, 241–252 (2009)
Lipowezky, U.: Selection of the optimal prototype subset for 1-NN classification. Pattern Recognit. Lett. 19, 907918 (1998). [Online]. Available: http://linkinghub.elsevier.com/retrieve/pii/S0167865598000750
Lum, K., Powell, J., Hihn, J.: Validation of spacecraft software cost estimation models for flight and ground systems. In: ISPA Conference Proceedings, Software Modeling Track, May 2002
Mann, H.B., Whitney, D.R.: On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18(1), 50–60 (1947). Available on-line at http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aoms/1177730491
Mendes, E., Watson, I.D., Triggs, C., Mosley, N., Counsell, S.: A comparative study of cost estimation models for web hypermedia applications. Empir. Softw. Eng. 8(2), 163–196 (2003)
Menzies, T., Port, D., Chen, Z., Hihn, J., Stukes, S.: Validation methods for calibrating software effort models. In: Proceedings, ICSE. Available from http://menzies.us/pdf/04coconut.pdf, (2005)
Menzies, T., Chen, Z., Hihn, J., Lum, K.: Selecting Best Practices for Effort Estimation. IEEE Trans. Softw. Eng. 32, 883–895 (2006)
Menzies, T., Chen, Z., Hihn, J., Lum, K.: Selecting best practices for effort estimation. IEEE Trans. Softw. Eng. 32(11), 883–895 (2006). Available from http://menzies.us/pdf/06coseekmo.pdf
Menzies, T., Chen, Z., Port, D., Hihn, J.: Simple software cost estimation: Safe or unsafe? In: Proceedings, PROMISE Workshop, ICSE 2005. Available from http://menzies.us/pdf/05safewhen.pdf
Miller, A.: Subset Selection in Regression, 2nd edn. Chapman & Hall, London (2002)
Myrtveit, I., Stensrud, E., Shepperd, M.: Reliability and validity in comparative studies of software prediction models. IEEE Trans. Softw. Eng. 31(5), 380–391 (2005)
Park, R.: The central equations of the price software cost model. In: 4th COCOMO Users Group Meeting, November 1988
Putnam, L., Myers, W.: Measures for Excellence. Yourdon Press Computing Series (1992)
Quinlan, J.R.: Learning with continuous classes. In: 5th Australian Joint Conference on Artificial Intelligence, pp. 343–348 (1992). Available from http://citeseer.nj.nec.com/quinlan92learning.html
Shepperd, M.: Software project economics: a roadmap. In: International Conference on Software Engineering 2007: Future of Software Engineering (2007)
Shepperd, M., Kadoda, G.F.: Comparing software prediction techniques using simulation. IEEE Trans. Softw. Eng. 27(11), 1014–1022 (2001)
Shepperd, M., Schofield, C.: Estimating software project effort using analogies. IEEE Trans. Softw. Eng. 23(11), 736–743 (1997). Available from http://www.utdallas.edu/~rbanker/SE_XII.pdf
Simons, D., Chabris, C.: Gorillas in our midst: sustained in attentional blindless for dynamic events perception. Perception 28, 1059–1074 (1999)
Spareref.com: NASA to shut down checkout & launch control system. August 26, 2002. http://www.spaceref.com/news/viewnews.html?id=475 (2002)
Walkerden, F., Jeffery, R.: An empirical study of analogy-based software effort estimation. Empir. Softw. Eng. 4(2), 135–158 (1999)
Witten, I.H., Frank, E.: Data mining, 2nd edn. Morgan Kaufmann, Los Altos (2005)
Yang, Y., Webb, G.I.: A comparative study of discretization methods for Naive-Bayes classifiers. In: Proceedings of PKAW 2002: The 2002 Pacific Rim Knowledge Acquisition Workshop, pp. 159–173 (2002)
Author information
Authors and Affiliations
Corresponding author
Additional information
The research described in this paper was carried out at West Virginia University and the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the US National Aeronautics and Space Administration. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not constitute or imply its endorsement by the US Government.
Rights and permissions
About this article
Cite this article
Menzies, T., Jalali, O., Hihn, J. et al. Stable rankings for different effort models. Autom Softw Eng 17, 409–437 (2010). https://doi.org/10.1007/s10515-010-0070-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10515-010-0070-z