ABSTRACT
Background: Continuously calibrated and validated parametric models are necessary for realistic software estimates. However, in practice, variations in model adoption and usage patterns introduce a great deal of local bias in the resultant historical data. Such local bias should be carefully examined and addressed before the historical data can be used for calibrating new versions of parametric models.
Aims: In this study, we aim at investigating the degree of such local bias in a cross-company historical dataset, and assessing its impacts on parametric estimation model's performance.
Method: Our study consists of three parts: 1) defining a method for measuring and analyzing the local bias associated with individual organization data subset in the overall dataset; 2) assessing the impacts of local bias on the estimation performance of COCOMO II 2000 model; 3) performing a correlation analysis to verify that local bias can be harmful to the performance of a parametric estimation model.
Results: Our results show that the local bias negatively impacts the performance of parametric model. Our measure of local bias has a positive correlation with the performance by statistical importance.
Conclusion: Local calibration by using the whole multi-company data would get worse performance. The influence of multi-company data could be defined by local bias and be measured by our method.
- Boehm, B. W., Clark, and Horowitz et al. Software Cost Estimation with Cocomo II with Cdrom: Prentice Hall PTR, 2000. Google ScholarDigital Library
- SEER-SEM Version 51 and Later User's Manual, Galorath Incorporated, March 2000 update.Google Scholar
- Your Guide to PRICES: Estimating Cost and Schedule of Software Development and Support, Mt. Laurel, New Jersey, PRICE Systems, LLC: 1998.Google Scholar
- NASA Cost Estimation handbook, http://www.ceh.nasa.gov/webhelpfiles/Cost_Estimating_Handbook_NASA_2004.htmGoogle Scholar
- Handbook for Software Cost Estimation (JPL). http://www.ceh.nasa.gov/downloadfiles/Web%20Links/cost_hb_public-6-5.pdfGoogle Scholar
- Stutzke, R. D. Estimating Software-Intensive Systems: Projects, Products, and Processes. Addison-Wesley Professional (May 6, 2005). Google ScholarDigital Library
- Chen, J., Yang, Y., and Nguyen, V. et al. Reducing the Local Bias in Calibrating the General COCOMO. 24th International Forum on COCOMO and Systems/Software Cost Modeling. Nov. 2-5, 2009. Cambridge, USA.Google Scholar
- Bottou, L., and Bousquet, O., The Tradeoffs of Large Scale Learning. Advances in Neural Information Processing Systems, 20:161--168, 2008Google Scholar
- Hastie, T., Tibshirani, R., and Friedman, J., The elements of statistical learning data mining, inference, and prediction. Springer; Corrected edition. July 30, 2003.Google Scholar
- Wasserman, L. All of Statistics: A Concise Course in Statistical Inference. Springer-Verlag, Berlin. 2003. Google ScholarDigital Library
- Kitchenham, B. A., Mendes, E., and Travassos, G. H. "Cross versus within-company cost estimation studies: A systematic review," IEEE Transactions on Software Engineering, vol. 33, no. 5, pp. 316--329, May, 2007. Google ScholarDigital Library
- Jeffery, R., Ruhe, M., and Wieczorek, I. "A comparative study of two software development cost modeling techniques using multi-organizational and company-specific data," Information and Software Technology, vol. 42, no. 14, pp. 1009--1016, Nov, 2000.Google ScholarCross Ref
- Kitchenham, B. "A procedure for analyzing unbalanced datasets," IEEE Transactions on Software Engineering, vol. 24, no. 4, pp. 278--301, Apr, 1998. Google ScholarDigital Library
- Liu, Q., and Mintram, R. "Preliminary data analysis methods in software estimation," Software Quality Journal, vol. 13, no. 1, pp. 91--115, Mar, 2005. Google ScholarDigital Library
- Cuadrado-Gallego, J. J., and Sicilia, M. A. "An algorithm for the generation of segmented parametric software estimation models and its empirical evaluation," Computing and Informatics, vol. 26, no. 1, pp. 1--15, 2007.Google Scholar
- Clark, B., Devnani-Chulani, S. and Boehm B. W. et al., "Calibrating the COCOMO II Post-Architecture model," Proceedings of the 1998 International Conference on Software Engineering, International Conference on Software Engineering, pp. 477--480, Los Alamitos: IEEE Computer Soc, 1998. Google ScholarDigital Library
- Nguyen, V., Steece, B., and Boehm, B. W. et al., A Constrained Regression Technique for COCOMO Calibration, New York: Assoc Computing Machinery, 2008.Google Scholar
- Chulani, S., Boehm, B. W., and Steece, B. "Bayesian analysis of empirical software engineering cost models," IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 573--583, Jul-Aug, 1999. Google ScholarDigital Library
- Yang, Y., Clark, B. COCOMO II. 2004 Calibration Status. 19th International Forum on COCOMO and Systems/Software Cost Modeling. Oct. 22-25, 2004. Los Angeles, USA.Google Scholar
- Menzies, T., Hihn, J. Evidence-Based Cost Estimation for Better Quality Software. IEEE Software. July/August 2006. Google ScholarDigital Library
- Menzies, T., Chen, Z., and Hihnet, J. et al. Selecting Best Practices for Effort Estimation. IEEE Transactions on Software Engineering. November 2006. Google ScholarDigital Library
- Xie, L., Yang, Y., and Yang, D. et al. Mean-Variance Combination (MVC): A New Method for Evaluating Effort Estimation Models. Accepted by the Symposium in Honor of Dr. Barry Boehm. April 25-26, 2011. Beijing China.Google Scholar
- Kocaguneli, E., Menzies, T., and Bener, A. et al. "Exploiting the Essential Assumptions of Analogy-Based Effort Estimation," IEEE Transactions on Software Engineering, 02 Mar. 2011. IEEE computer Society Digital Library. IEEE Computer Society Google ScholarDigital Library
- Port, D., and Korte, M. (2008). Comparative Studies of the Model Evaluation Criterions MMRE and PRED on Software Cost Estimation Research. ACM, Liepzig, Germany, pp. 63--70.Google ScholarDigital Library
- Jørgensen, M., Evidence-based guidelines for assessment of software development cost uncertainty. Software Engineering, IEEE Transactions on, 2005. 31(11): p. 942--954. Google ScholarDigital Library
- Jørgensen, M., and Teigen, K. H. 2002. Uncertainty intervals versus interval uncertainty: An alternative method for eliciting effort prediction intervals in software development projects. International Conference on Project Management (ProMAC), Singapore, pp. 343--352.Google Scholar
- Jørgensen, M., Teigen, K. H., and Moløkken-østvold, K. J, 2004. Better sure than safe? Overconfidence in judgment based software development effort prediction intervals. Journal of Systems and Software 70(1-2): 79--93. Google ScholarDigital Library
Index Terms
Local bias and its impacts on the performance of parametric estimation models
Recommendations
Analyzing and handling local bias for calibrating parametric cost estimation models
ContextParametric cost estimation models need to be continuously calibrated and improved to assure more accurate software estimates and reflect changing software development contexts. Local calibration by tuning a subset of model parameters is a ...
Improved measurement of software development effort estimation bias
Highlights- An effort estimate is the estimate of a point in an effort probability distribution.
Abstract ContextWhile prior software development effort estimation research has examined the properties of estimation error measures, there has not been much research on the properties of measures of estimation bias.
...Software Effort Estimation of GSD Projects Using Calibrated Parametric Estimation Models
ICTCS '16: Proceedings of the Second International Conference on Information and Communication Technology for Competitive StrategiesSoftware Engineering is the domain developed for designing, coding and testing of various software projects of computer and other electronic devices. GSD is the environment for developing projects at geographically isolated areas beyond cultural ...
Comments