skip to main content
10.1145/2020390.2020404acmotherconferencesArticle/Chapter ViewAbstractPublication PagespromiseConference Proceedingsconference-collections
research-article

Local bias and its impacts on the performance of parametric estimation models

Published: 20 September 2011 Publication History

Abstract

Background: Continuously calibrated and validated parametric models are necessary for realistic software estimates. However, in practice, variations in model adoption and usage patterns introduce a great deal of local bias in the resultant historical data. Such local bias should be carefully examined and addressed before the historical data can be used for calibrating new versions of parametric models.
Aims: In this study, we aim at investigating the degree of such local bias in a cross-company historical dataset, and assessing its impacts on parametric estimation model's performance.
Method: Our study consists of three parts: 1) defining a method for measuring and analyzing the local bias associated with individual organization data subset in the overall dataset; 2) assessing the impacts of local bias on the estimation performance of COCOMO II 2000 model; 3) performing a correlation analysis to verify that local bias can be harmful to the performance of a parametric estimation model.
Results: Our results show that the local bias negatively impacts the performance of parametric model. Our measure of local bias has a positive correlation with the performance by statistical importance.
Conclusion: Local calibration by using the whole multi-company data would get worse performance. The influence of multi-company data could be defined by local bias and be measured by our method.

References

[1]
Boehm, B. W., Clark, and Horowitz et al. Software Cost Estimation with Cocomo II with Cdrom: Prentice Hall PTR, 2000.
[2]
SEER-SEM Version 51 and Later User's Manual, Galorath Incorporated, March 2000 update.
[3]
Your Guide to PRICES: Estimating Cost and Schedule of Software Development and Support, Mt. Laurel, New Jersey, PRICE Systems, LLC: 1998.
[4]
NASA Cost Estimation handbook, http://www.ceh.nasa.gov/webhelpfiles/Cost_Estimating_Handbook_NASA_2004.htm
[5]
Handbook for Software Cost Estimation (JPL). http://www.ceh.nasa.gov/downloadfiles/Web%20Links/cost_hb_public-6-5.pdf
[6]
Stutzke, R. D. Estimating Software-Intensive Systems: Projects, Products, and Processes. Addison-Wesley Professional (May 6, 2005).
[7]
Chen, J., Yang, Y., and Nguyen, V. et al. Reducing the Local Bias in Calibrating the General COCOMO. 24th International Forum on COCOMO and Systems/Software Cost Modeling. Nov. 2-5, 2009. Cambridge, USA.
[8]
Bottou, L., and Bousquet, O., The Tradeoffs of Large Scale Learning. Advances in Neural Information Processing Systems, 20:161--168, 2008
[9]
Hastie, T., Tibshirani, R., and Friedman, J., The elements of statistical learning data mining, inference, and prediction. Springer; Corrected edition. July 30, 2003.
[10]
Wasserman, L. All of Statistics: A Concise Course in Statistical Inference. Springer-Verlag, Berlin. 2003.
[11]
Kitchenham, B. A., Mendes, E., and Travassos, G. H. "Cross versus within-company cost estimation studies: A systematic review," IEEE Transactions on Software Engineering, vol. 33, no. 5, pp. 316--329, May, 2007.
[12]
Jeffery, R., Ruhe, M., and Wieczorek, I. "A comparative study of two software development cost modeling techniques using multi-organizational and company-specific data," Information and Software Technology, vol. 42, no. 14, pp. 1009--1016, Nov, 2000.
[13]
Kitchenham, B. "A procedure for analyzing unbalanced datasets," IEEE Transactions on Software Engineering, vol. 24, no. 4, pp. 278--301, Apr, 1998.
[14]
Liu, Q., and Mintram, R. "Preliminary data analysis methods in software estimation," Software Quality Journal, vol. 13, no. 1, pp. 91--115, Mar, 2005.
[15]
Cuadrado-Gallego, J. J., and Sicilia, M. A. "An algorithm for the generation of segmented parametric software estimation models and its empirical evaluation," Computing and Informatics, vol. 26, no. 1, pp. 1--15, 2007.
[16]
Clark, B., Devnani-Chulani, S. and Boehm B. W. et al., "Calibrating the COCOMO II Post-Architecture model," Proceedings of the 1998 International Conference on Software Engineering, International Conference on Software Engineering, pp. 477--480, Los Alamitos: IEEE Computer Soc, 1998.
[17]
Nguyen, V., Steece, B., and Boehm, B. W. et al., A Constrained Regression Technique for COCOMO Calibration, New York: Assoc Computing Machinery, 2008.
[18]
Chulani, S., Boehm, B. W., and Steece, B. "Bayesian analysis of empirical software engineering cost models," IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 573--583, Jul-Aug, 1999.
[19]
Yang, Y., Clark, B. COCOMO II. 2004 Calibration Status. 19th International Forum on COCOMO and Systems/Software Cost Modeling. Oct. 22-25, 2004. Los Angeles, USA.
[20]
Menzies, T., Hihn, J. Evidence-Based Cost Estimation for Better Quality Software. IEEE Software. July/August 2006.
[21]
Menzies, T., Chen, Z., and Hihnet, J. et al. Selecting Best Practices for Effort Estimation. IEEE Transactions on Software Engineering. November 2006.
[22]
Xie, L., Yang, Y., and Yang, D. et al. Mean-Variance Combination (MVC): A New Method for Evaluating Effort Estimation Models. Accepted by the Symposium in Honor of Dr. Barry Boehm. April 25-26, 2011. Beijing China.
[23]
Kocaguneli, E., Menzies, T., and Bener, A. et al. "Exploiting the Essential Assumptions of Analogy-Based Effort Estimation," IEEE Transactions on Software Engineering, 02 Mar. 2011. IEEE computer Society Digital Library. IEEE Computer Society
[24]
Port, D., and Korte, M. (2008). Comparative Studies of the Model Evaluation Criterions MMRE and PRED on Software Cost Estimation Research. ACM, Liepzig, Germany, pp. 63--70.
[25]
Jørgensen, M., Evidence-based guidelines for assessment of software development cost uncertainty. Software Engineering, IEEE Transactions on, 2005. 31(11): p. 942--954.
[26]
Jørgensen, M., and Teigen, K. H. 2002. Uncertainty intervals versus interval uncertainty: An alternative method for eliciting effort prediction intervals in software development projects. International Conference on Project Management (ProMAC), Singapore, pp. 343--352.
[27]
Jørgensen, M., Teigen, K. H., and Moløkken-østvold, K. J, 2004. Better sure than safe? Overconfidence in judgment based software development effort prediction intervals. Journal of Systems and Software 70(1-2): 79--93.

Cited By

View all
  • (2019)Bellwethers: A Baseline Method for Transfer LearningIEEE Transactions on Software Engineering10.1109/TSE.2018.282167045:11(1081-1105)Online publication date: 1-Nov-2019
  • (2018)COSMIC Function Points Evaluation for Software MaintenanceProceedings of the 11th Innovations in Software Engineering Conference10.1145/3172871.3172874(1-11)Online publication date: 9-Feb-2018
  • (2018)Heterogeneous Defect PredictionIEEE Transactions on Software Engineering10.1109/TSE.2017.272060344:9(874-896)Online publication date: 1-Sep-2018
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
Promise '11: Proceedings of the 7th International Conference on Predictive Models in Software Engineering
September 2011
145 pages
ISBN:9781450307093
DOI:10.1145/2020390
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 September 2011

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. accuracy indicator
  2. effort estimation
  3. local bias
  4. parametric model

Qualifiers

  • Research-article

Funding Sources

Conference

Promise '11

Acceptance Rates

Promise '11 Paper Acceptance Rate 15 of 35 submissions, 43%;
Overall Acceptance Rate 98 of 213 submissions, 46%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)8
  • Downloads (Last 6 weeks)1
Reflects downloads up to 25 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2019)Bellwethers: A Baseline Method for Transfer LearningIEEE Transactions on Software Engineering10.1109/TSE.2018.282167045:11(1081-1105)Online publication date: 1-Nov-2019
  • (2018)COSMIC Function Points Evaluation for Software MaintenanceProceedings of the 11th Innovations in Software Engineering Conference10.1145/3172871.3172874(1-11)Online publication date: 9-Feb-2018
  • (2018)Heterogeneous Defect PredictionIEEE Transactions on Software Engineering10.1109/TSE.2017.272060344:9(874-896)Online publication date: 1-Sep-2018
  • (2016)Calibrating COCOMO® II for projects with high personnel turnoverProceedings of the International Conference on Software and Systems Process10.1145/2904354.2904367(51-55)Online publication date: 14-May-2016
  • (2015)Transfer learning in effort estimationEmpirical Software Engineering10.1007/s10664-014-9300-520:3(813-843)Online publication date: 1-Jun-2015
  • (2013)Local versus Global Lessons for Defect Prediction and Effort EstimationIEEE Transactions on Software Engineering10.1109/TSE.2012.8339:6(822-834)Online publication date: 1-Jun-2013
  • (2013)Analyzing and handling local bias for calibrating parametric cost estimation modelsInformation and Software Technology10.1016/j.infsof.2013.03.00255:8(1496-1511)Online publication date: 1-Aug-2013

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media