research-article

Local bias and its impacts on the performance of parametric estimation models

Authors:

Ricardo ValerdiAuthors Info & Claims

Promise '11: Proceedings of the 7th International Conference on Predictive Models in Software Engineering

Article No.: 14, Pages 1 - 10

https://doi.org/10.1145/2020390.2020404

Published: 20 September 2011 Publication History

Abstract

Background: Continuously calibrated and validated parametric models are necessary for realistic software estimates. However, in practice, variations in model adoption and usage patterns introduce a great deal of local bias in the resultant historical data. Such local bias should be carefully examined and addressed before the historical data can be used for calibrating new versions of parametric models.

Aims: In this study, we aim at investigating the degree of such local bias in a cross-company historical dataset, and assessing its impacts on parametric estimation model's performance.

Method: Our study consists of three parts: 1) defining a method for measuring and analyzing the local bias associated with individual organization data subset in the overall dataset; 2) assessing the impacts of local bias on the estimation performance of COCOMO II 2000 model; 3) performing a correlation analysis to verify that local bias can be harmful to the performance of a parametric estimation model.

Results: Our results show that the local bias negatively impacts the performance of parametric model. Our measure of local bias has a positive correlation with the performance by statistical importance.

Conclusion: Local calibration by using the whole multi-company data would get worse performance. The influence of multi-company data could be defined by local bias and be measured by our method.

References

[1]

Boehm, B. W., Clark, and Horowitz et al. Software Cost Estimation with Cocomo II with Cdrom: Prentice Hall PTR, 2000.

Digital Library

[2]

SEER-SEM Version 51 and Later User's Manual, Galorath Incorporated, March 2000 update.

[3]

Your Guide to PRICES: Estimating Cost and Schedule of Software Development and Support, Mt. Laurel, New Jersey, PRICE Systems, LLC: 1998.

[4]

NASA Cost Estimation handbook, http://www.ceh.nasa.gov/webhelpfiles/Cost_Estimating_Handbook_NASA_2004.htm

[5]

Handbook for Software Cost Estimation (JPL). http://www.ceh.nasa.gov/downloadfiles/Web%20Links/cost_hb_public-6-5.pdf

[6]

Stutzke, R. D. Estimating Software-Intensive Systems: Projects, Products, and Processes. Addison-Wesley Professional (May 6, 2005).

Digital Library

[7]

Chen, J., Yang, Y., and Nguyen, V. et al. Reducing the Local Bias in Calibrating the General COCOMO. 24th International Forum on COCOMO and Systems/Software Cost Modeling. Nov. 2-5, 2009. Cambridge, USA.

[8]

Bottou, L., and Bousquet, O., The Tradeoffs of Large Scale Learning. Advances in Neural Information Processing Systems, 20:161--168, 2008

[9]

Hastie, T., Tibshirani, R., and Friedman, J., The elements of statistical learning data mining, inference, and prediction. Springer; Corrected edition. July 30, 2003.

[10]

Wasserman, L. All of Statistics: A Concise Course in Statistical Inference. Springer-Verlag, Berlin. 2003.

Digital Library

[11]

Kitchenham, B. A., Mendes, E., and Travassos, G. H. "Cross versus within-company cost estimation studies: A systematic review," IEEE Transactions on Software Engineering, vol. 33, no. 5, pp. 316--329, May, 2007.

Digital Library

[12]

Jeffery, R., Ruhe, M., and Wieczorek, I. "A comparative study of two software development cost modeling techniques using multi-organizational and company-specific data," Information and Software Technology, vol. 42, no. 14, pp. 1009--1016, Nov, 2000.

[13]

Kitchenham, B. "A procedure for analyzing unbalanced datasets," IEEE Transactions on Software Engineering, vol. 24, no. 4, pp. 278--301, Apr, 1998.

Digital Library

[14]

Liu, Q., and Mintram, R. "Preliminary data analysis methods in software estimation," Software Quality Journal, vol. 13, no. 1, pp. 91--115, Mar, 2005.

Digital Library

[15]

Cuadrado-Gallego, J. J., and Sicilia, M. A. "An algorithm for the generation of segmented parametric software estimation models and its empirical evaluation," Computing and Informatics, vol. 26, no. 1, pp. 1--15, 2007.

[16]

Clark, B., Devnani-Chulani, S. and Boehm B. W. et al., "Calibrating the COCOMO II Post-Architecture model," Proceedings of the 1998 International Conference on Software Engineering, International Conference on Software Engineering, pp. 477--480, Los Alamitos: IEEE Computer Soc, 1998.

Digital Library

[17]

Nguyen, V., Steece, B., and Boehm, B. W. et al., A Constrained Regression Technique for COCOMO Calibration, New York: Assoc Computing Machinery, 2008.

[18]

Chulani, S., Boehm, B. W., and Steece, B. "Bayesian analysis of empirical software engineering cost models," IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 573--583, Jul-Aug, 1999.

Digital Library

[19]

Yang, Y., Clark, B. COCOMO II. 2004 Calibration Status. 19th International Forum on COCOMO and Systems/Software Cost Modeling. Oct. 22-25, 2004. Los Angeles, USA.

[20]

Menzies, T., Hihn, J. Evidence-Based Cost Estimation for Better Quality Software. IEEE Software. July/August 2006.

Digital Library

[21]

Menzies, T., Chen, Z., and Hihnet, J. et al. Selecting Best Practices for Effort Estimation. IEEE Transactions on Software Engineering. November 2006.

Digital Library

[22]

Xie, L., Yang, Y., and Yang, D. et al. Mean-Variance Combination (MVC): A New Method for Evaluating Effort Estimation Models. Accepted by the Symposium in Honor of Dr. Barry Boehm. April 25-26, 2011. Beijing China.

[23]

Kocaguneli, E., Menzies, T., and Bener, A. et al. "Exploiting the Essential Assumptions of Analogy-Based Effort Estimation," IEEE Transactions on Software Engineering, 02 Mar. 2011. IEEE computer Society Digital Library. IEEE Computer Society

Digital Library

[24]

Port, D., and Korte, M. (2008). Comparative Studies of the Model Evaluation Criterions MMRE and PRED on Software Cost Estimation Research. ACM, Liepzig, Germany, pp. 63--70.

Digital Library

[25]

Jørgensen, M., Evidence-based guidelines for assessment of software development cost uncertainty. Software Engineering, IEEE Transactions on, 2005. 31(11): p. 942--954.

Digital Library

[26]

Jørgensen, M., and Teigen, K. H. 2002. Uncertainty intervals versus interval uncertainty: An alternative method for eliciting effort prediction intervals in software development projects. International Conference on Project Management (ProMAC), Singapore, pp. 343--352.

[27]

Jørgensen, M., Teigen, K. H., and Moløkken-østvold, K. J, 2004. Better sure than safe? Overconfidence in judgment based software development effort prediction intervals. Journal of Systems and Software 70(1-2): 79--93.

Digital Library

Cited By

Krishna RMenzies T(2019)Bellwethers: A Baseline Method for Transfer LearningIEEE Transactions on Software Engineering10.1109/TSE.2018.282167045:11(1081-1105)Online publication date: 1-Nov-2019
https://doi.org/10.1109/TSE.2018.2821670
Hira ABoehm BReddy YVarma VCleland JBellur USengupta SSharma NLoganathan RSharma RSarkar S(2018)COSMIC Function Points Evaluation for Software MaintenanceProceedings of the 11th Innovations in Software Engineering Conference10.1145/3172871.3172874(1-11)Online publication date: 9-Feb-2018
https://dl.acm.org/doi/10.1145/3172871.3172874
Nam JFu WKim SMenzies TTan L(2018)Heterogeneous Defect PredictionIEEE Transactions on Software Engineering10.1109/TSE.2017.272060344:9(874-896)Online publication date: 1-Sep-2018
https://doi.org/10.1109/TSE.2017.2720603
Show More Cited By

Index Terms

Local bias and its impacts on the performance of parametric estimation models
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Implementation management
        Pricing and resource allocation

Recommendations

Analyzing and handling local bias for calibrating parametric cost estimation models

ContextParametric cost estimation models need to be continuously calibrated and improved to assure more accurate software estimates and reflect changing software development contexts. Local calibration by tuning a subset of model parameters is a frequent ...
Improved measurement of software development effort estimation bias
Highlights
- An effort estimate is the estimate of a point in an effort probability distribution.
Abstract Context
While prior software development effort estimation research has examined the properties of estimation error measures, there has not been much research on the properties of measures of estimation bias.
...
Software Effort Estimation of GSD Projects Using Calibrated Parametric Estimation Models
ICTCS '16: Proceedings of the Second International Conference on Information and Communication Technology for Competitive Strategies

Software Engineering is the domain developed for designing, coding and testing of various software projects of computer and other electronic devices. GSD is the environment for developing projects at geographically isolated areas beyond cultural ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

Promise '11: Proceedings of the 7th International Conference on Predictive Models in Software Engineering

September 2011

145 pages

ISBN:9781450307093

DOI:10.1145/2020390

General Chair:
Tim Menzies
WVU

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 September 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

Promise '11

Promise '11: 7th International Conference on Predictive Models in Software Engineering

September 20 - 21, 2011

Alberta, Banff, Canada

Acceptance Rates

Promise '11 Paper Acceptance Rate 15 of 35 submissions, 43%;

Overall Acceptance Rate 98 of 213 submissions, 46%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
235
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Krishna RMenzies T(2019)Bellwethers: A Baseline Method for Transfer LearningIEEE Transactions on Software Engineering10.1109/TSE.2018.282167045:11(1081-1105)Online publication date: 1-Nov-2019
https://doi.org/10.1109/TSE.2018.2821670
Hira ABoehm BReddy YVarma VCleland JBellur USengupta SSharma NLoganathan RSharma RSarkar S(2018)COSMIC Function Points Evaluation for Software MaintenanceProceedings of the 11th Innovations in Software Engineering Conference10.1145/3172871.3172874(1-11)Online publication date: 9-Feb-2018
https://dl.acm.org/doi/10.1145/3172871.3172874
Nam JFu WKim SMenzies TTan L(2018)Heterogeneous Defect PredictionIEEE Transactions on Software Engineering10.1109/TSE.2017.272060344:9(874-896)Online publication date: 1-Sep-2018
https://doi.org/10.1109/TSE.2017.2720603
Hira ASharma SBoehm BPerry DRaffo D(2016)Calibrating COCOMO® II for projects with high personnel turnoverProceedings of the International Conference on Software and Systems Process10.1145/2904354.2904367(51-55)Online publication date: 14-May-2016
https://dl.acm.org/doi/10.1145/2904354.2904367
Kocaguneli EMenzies TMendes E(2015)Transfer learning in effort estimationEmpirical Software Engineering10.1007/s10664-014-9300-520:3(813-843)Online publication date: 1-Jun-2015
https://dl.acm.org/doi/10.1007/s10664-014-9300-5
Menzies TButcher ACok DMarcus ALayman LShull FTurhan BZimmermann T(2013)Local versus Global Lessons for Defect Prediction and Effort EstimationIEEE Transactions on Software Engineering10.1109/TSE.2012.8339:6(822-834)Online publication date: 1-Jun-2013
https://dl.acm.org/doi/10.1109/TSE.2012.83
Yang YHe ZMao KLi QNguyen VBoehm BValerdi R(2013)Analyzing and handling local bias for calibrating parametric cost estimation modelsInformation and Software Technology10.1016/j.infsof.2013.03.00255:8(1496-1511)Online publication date: 1-Aug-2013
https://dl.acm.org/doi/10.1016/j.infsof.2013.03.002

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten