skip to main content
10.1145/2020390.2020404acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

Local bias and its impacts on the performance of parametric estimation models

Authors Info & Claims
Published:20 September 2011Publication History

ABSTRACT

Background: Continuously calibrated and validated parametric models are necessary for realistic software estimates. However, in practice, variations in model adoption and usage patterns introduce a great deal of local bias in the resultant historical data. Such local bias should be carefully examined and addressed before the historical data can be used for calibrating new versions of parametric models.

Aims: In this study, we aim at investigating the degree of such local bias in a cross-company historical dataset, and assessing its impacts on parametric estimation model's performance.

Method: Our study consists of three parts: 1) defining a method for measuring and analyzing the local bias associated with individual organization data subset in the overall dataset; 2) assessing the impacts of local bias on the estimation performance of COCOMO II 2000 model; 3) performing a correlation analysis to verify that local bias can be harmful to the performance of a parametric estimation model.

Results: Our results show that the local bias negatively impacts the performance of parametric model. Our measure of local bias has a positive correlation with the performance by statistical importance.

Conclusion: Local calibration by using the whole multi-company data would get worse performance. The influence of multi-company data could be defined by local bias and be measured by our method.

References

  1. Boehm, B. W., Clark, and Horowitz et al. Software Cost Estimation with Cocomo II with Cdrom: Prentice Hall PTR, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. SEER-SEM Version 51 and Later User's Manual, Galorath Incorporated, March 2000 update.Google ScholarGoogle Scholar
  3. Your Guide to PRICES: Estimating Cost and Schedule of Software Development and Support, Mt. Laurel, New Jersey, PRICE Systems, LLC: 1998.Google ScholarGoogle Scholar
  4. NASA Cost Estimation handbook, http://www.ceh.nasa.gov/webhelpfiles/Cost_Estimating_Handbook_NASA_2004.htmGoogle ScholarGoogle Scholar
  5. Handbook for Software Cost Estimation (JPL). http://www.ceh.nasa.gov/downloadfiles/Web%20Links/cost_hb_public-6-5.pdfGoogle ScholarGoogle Scholar
  6. Stutzke, R. D. Estimating Software-Intensive Systems: Projects, Products, and Processes. Addison-Wesley Professional (May 6, 2005). Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Chen, J., Yang, Y., and Nguyen, V. et al. Reducing the Local Bias in Calibrating the General COCOMO. 24th International Forum on COCOMO and Systems/Software Cost Modeling. Nov. 2-5, 2009. Cambridge, USA.Google ScholarGoogle Scholar
  8. Bottou, L., and Bousquet, O., The Tradeoffs of Large Scale Learning. Advances in Neural Information Processing Systems, 20:161--168, 2008Google ScholarGoogle Scholar
  9. Hastie, T., Tibshirani, R., and Friedman, J., The elements of statistical learning data mining, inference, and prediction. Springer; Corrected edition. July 30, 2003.Google ScholarGoogle Scholar
  10. Wasserman, L. All of Statistics: A Concise Course in Statistical Inference. Springer-Verlag, Berlin. 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Kitchenham, B. A., Mendes, E., and Travassos, G. H. "Cross versus within-company cost estimation studies: A systematic review," IEEE Transactions on Software Engineering, vol. 33, no. 5, pp. 316--329, May, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Jeffery, R., Ruhe, M., and Wieczorek, I. "A comparative study of two software development cost modeling techniques using multi-organizational and company-specific data," Information and Software Technology, vol. 42, no. 14, pp. 1009--1016, Nov, 2000.Google ScholarGoogle ScholarCross RefCross Ref
  13. Kitchenham, B. "A procedure for analyzing unbalanced datasets," IEEE Transactions on Software Engineering, vol. 24, no. 4, pp. 278--301, Apr, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Liu, Q., and Mintram, R. "Preliminary data analysis methods in software estimation," Software Quality Journal, vol. 13, no. 1, pp. 91--115, Mar, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Cuadrado-Gallego, J. J., and Sicilia, M. A. "An algorithm for the generation of segmented parametric software estimation models and its empirical evaluation," Computing and Informatics, vol. 26, no. 1, pp. 1--15, 2007.Google ScholarGoogle Scholar
  16. Clark, B., Devnani-Chulani, S. and Boehm B. W. et al., "Calibrating the COCOMO II Post-Architecture model," Proceedings of the 1998 International Conference on Software Engineering, International Conference on Software Engineering, pp. 477--480, Los Alamitos: IEEE Computer Soc, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Nguyen, V., Steece, B., and Boehm, B. W. et al., A Constrained Regression Technique for COCOMO Calibration, New York: Assoc Computing Machinery, 2008.Google ScholarGoogle Scholar
  18. Chulani, S., Boehm, B. W., and Steece, B. "Bayesian analysis of empirical software engineering cost models," IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 573--583, Jul-Aug, 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Yang, Y., Clark, B. COCOMO II. 2004 Calibration Status. 19th International Forum on COCOMO and Systems/Software Cost Modeling. Oct. 22-25, 2004. Los Angeles, USA.Google ScholarGoogle Scholar
  20. Menzies, T., Hihn, J. Evidence-Based Cost Estimation for Better Quality Software. IEEE Software. July/August 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Menzies, T., Chen, Z., and Hihnet, J. et al. Selecting Best Practices for Effort Estimation. IEEE Transactions on Software Engineering. November 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Xie, L., Yang, Y., and Yang, D. et al. Mean-Variance Combination (MVC): A New Method for Evaluating Effort Estimation Models. Accepted by the Symposium in Honor of Dr. Barry Boehm. April 25-26, 2011. Beijing China.Google ScholarGoogle Scholar
  23. Kocaguneli, E., Menzies, T., and Bener, A. et al. "Exploiting the Essential Assumptions of Analogy-Based Effort Estimation," IEEE Transactions on Software Engineering, 02 Mar. 2011. IEEE computer Society Digital Library. IEEE Computer Society Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Port, D., and Korte, M. (2008). Comparative Studies of the Model Evaluation Criterions MMRE and PRED on Software Cost Estimation Research. ACM, Liepzig, Germany, pp. 63--70.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Jørgensen, M., Evidence-based guidelines for assessment of software development cost uncertainty. Software Engineering, IEEE Transactions on, 2005. 31(11): p. 942--954. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Jørgensen, M., and Teigen, K. H. 2002. Uncertainty intervals versus interval uncertainty: An alternative method for eliciting effort prediction intervals in software development projects. International Conference on Project Management (ProMAC), Singapore, pp. 343--352.Google ScholarGoogle Scholar
  27. Jørgensen, M., Teigen, K. H., and Moløkken-østvold, K. J, 2004. Better sure than safe? Overconfidence in judgment based software development effort prediction intervals. Journal of Systems and Software 70(1-2): 79--93. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Local bias and its impacts on the performance of parametric estimation models

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Other conferences
      Promise '11: Proceedings of the 7th International Conference on Predictive Models in Software Engineering
      September 2011
      145 pages
      ISBN:9781450307093
      DOI:10.1145/2020390

      Copyright © 2011 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 20 September 2011

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

      Acceptance Rates

      Overall Acceptance Rate64of125submissions,51%

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader