Abstract
Traditionally, Software Effort Estimation (SEE) has been portrayed as a technical prediction task, for which we seek accuracy through improved estimation methods and a thorough consideration of effort predictors. In this article, our objective to make explicit the perspective of SEE as a behavioral act, bringing attention to the fact that human biases and noise are relevant components in estimation errors, acknowledging that SEE is more than a prediction task. We employed a thematic analysis of factors affecting expert judgment software estimates to satisfy this objective. We show that estimators do not necessarily behave entirely rationally given the information they have as input for estimation. The reception of estimation requests, the communication of software estimates, and their use also impact the estimation values — something unexpected if estimators were solely focused on SEE as a prediction task. Based on this, we also matched SEE interventions to behavioral ones from Behavioral Economics showing that, although we are already adopting behavioral insights to improve our estimation practices, there are still gaps to build upon. Furthermore, we assessed the strength of evidence for each of our review findings to derive recommendations for practitioners on the SEE interventions they can confidently adopt to improve their estimation processes. Moreover, in assessing the strength of evidence, we adopted the GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative research) approach. It enabled us to point concrete research paths to strengthen the existing evidence about SEE interventions based on the dimensions of the GRADE-CERQual evaluation scheme.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
All material generated during the current study is available at Figshare (https://doi.org/10.6084/m9.figshare.19406945.v1), as we describe here: − Online Resource 1 presents the relationships between factors and latent themes. − Online Resource 2 presents the codebook with the categories, general and specific strategies, and their descriptions, composing the analytical framework. − Online Resource 3 presents the list of papers included in the current study, along with the Evidence Profile and Summary of Qualitative Findings Tables. − Online Resource 4 presents the quality assessment for each paper we included in the current study.
Notes
However, Halkjelsvik and Jørgensen (2018) do not mention noise explicitly in their discussion.
The SLM included 131 papers in total.
References
Alhamed M, Storer T (2021) Playing planning poker in crowds: human computation of software effort estimates. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), IEEE, Madrid, ES, pp 1–12. https://doi.org/10.1109/ICSE43902.2021.00014
Altaleb A, Gravell A (2019) An empirical investigation of effort estimation in mobile apps using agile development process. J Softw 14(8):356–369. http://www.jsoftware.us/index.php?m=content &c=index &a=show &catid=211 &id=2959
Altaleb A, Alhashimi H, Gravell A (2020) A case study validation of the pair-estimation technique in effort estimation of mobile app development using agile processes. In: 2020 10th international conference on advanced computer information technologies (ACIT), IEEE, Deggendorf, pp 469–473. https://doi.org/10.1109/ACIT49673.2020.9208985
Ampatzoglou A, Bibi S, Avgeriou P, Verbeek M, Chatzigeorgiou A (2019) Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Inf Softw Technol 106:201–230. https://doi.org/10.1016/j.infsof.2018.10.006, http://www.sciencedirect.com/science/article/pii/S0950584918302106
Andrews J, Guyatt G, Oxman AD, Alderson P, Dahm P, Falck-Ytter Y, Nasser M, Meerpohl J, Post PN, Kunz R, Brozek J, Vist G, Rind D, Akl EA, Schünemann HJ (2013a) GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations. J Clin Epidemiol 66(7):719–725, https://doi.org/10.1016/j.jclinepi.2012.03.013
Andrews JC, Schünemann HJ, Oxman AD, Pottie K, Meerpohl JJ, Coello PA, Rind D, Montori VM, Brito JP, Norris S, Elbarbary M, Post P, Nasser M, Shukla V, Jaeschke R, Brozek J, Djulbegovic B, Guyatt G (2013b) GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation’s direction and strength. J Clin Epidemiol 66(7):726–735, https://doi.org/10.1016/j.jclinepi.2013.02.003
Aranda J, Easterbrook S (2005) Anchoring and adjustment in software estimation. In: Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on foundations of software engineering, ESEC/FSE-13, ACM, New York, pp 346–355. https://doi.org/10.1145/1081706.1081761, http://doi.acm.org/10.1145/1081706.1081761
Arifin HH, Daengdej J, Khanh NT (2017) An empirical study of effort-size and effort-time in expert-based estimations. In: 2017 8th international workshop on empirical software engineering in practice (IWESEP), IEEE, Tokyo, pp 35–40, https://doi.org/10.1109/IWESEP.2017.21
Arnuphaptrairong T (2021) Enhancing Delphi method with algorithmic estimates for software effort estimation: an experimentalstudy. SSRN Scholarly Paper ID 3898965, Social Science Research Network, Rochester, https://papers.ssrn.com/abstract=3898965
Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77–101. https://doi.org/10.1191/1478088706qp063oa
Braun V, Clarke V (2021) One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qual Res Psychol 18(3):328–352. https://doi.org/10.1080/14780887.2020.1769238
Briggs RA (2019) Normative Theories of rational choice: expected utility. In: Zalta EN (ed) The stanford encyclopedia of philosophy, fall 2019 edn, Metaphysics Research Lab, Stanford University, https://plato.stanford.edu/archives/fall2019/entries/rationality-normative-utility/
Brooks F (1995) Mythical man-month, the: essays on software engineering, Anniversary Edition, anniversary, edition. Addison-Wesley Professional, Reading, Mass
Brzezicka J, Wisniewski R (2014) Homo Oeconomicus and behavioral economics. Contemp Econ 8(4):353–364. https://doi.org/10.5709/ce.1897-9254.150
Buyalskaya A, Gallo M, Camerer CF (2021) The golden age of social science. Proc Natl Acad Sci 118(5). https://doi.org/10.1073/pnas.2002923118
Cohn M (2005) Agile estimating and planning, 1st edn. Robert C, Martin Series, Pearson
Colvin CJ, Garside R, Wainwright M, Munthe-Kaas H, Glenton C, Bohren MA, Carlsen B, Tunçalp O, Noyes J, Booth A, Rashidian A, Flottorp S, Lewin S (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 4: how to assess coherence. Implementation Sci 13(1):13. https://doi.org/10.1186/s13012-017-0691-8
Connelly LM, Peltzer JN (2016) Underdeveloped themes in qualitative research: relationship with interviews and analysis. Clin Nurse Specialist CNS 30(1):52–57. https://doi.org/10.1097/NUR.0000000000000173
Conoscenti M, Besner V, Vetrò A, Fernández DM (2019) Combining data analytics and developers feedback for identifying reasons of inaccurate estimations in agile software development. J Syst Softw 156:126–135. https://doi.org/10.1016/j.jss.2019.06.075, . www.sciencedirect.com/science/article/pii/S0164121219301372
Cruzes DS, Dyba T (2011) Recommended steps for thematic synthesis in software engineering. In: Proceedings of the 2011 international symposiumon empirical software engineering and measurement, ESEM’11, IEEE Computer Society, Washington, pp 275–284, https://doi.org/10.1109/ESEM.2011.36
DeMarco T, Lister T, House D (2013) Peopleware: productive projects and teams, 3rd edn. Addison-Wesley Professional, Upper Saddle River
Dybøa T, Dingsøyr T (2008) Strength of evidence in systematic reviews in software engineering. In: Proceedings of the second ACM-IEEE international symposium on Empirical software engineering and measurement, ESEM ’08, Association for Computing Machinery, New York, pp 178–187, https://doi.org/10.1145/1414004.1414034
Fereday J, Muir-Cochrane E (2006) Demonstrating Rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development. Int J Qual Methods 5(1):80–92. https://doi.org/10.1177/160940690600500107
Fleischmann M, Amirpur M, Benlian A, Hess T (2014) Cognitive Biaes in information systems research: a scientometric analysis. ECIS 2014 proceedings. https://aisel.aisnet.org/ecis2014/proceedings/track02/5
Fox CR, Ratner RK, Lieb DS (2005) How subjective grouping of options influences choice and allocation: diversification bias and the phenomenon of partition dependence. J Exp Psychol Gener 134(4):538–551. https://doi.org/10.1037/0096-3445.134.4.538
Frid-Nielsen SS, Jensen MD (2021) Maps of behavioural economics: evidence from the field. J Interdiscipl Econ 33(2):226–250. https://doi.org/10.1177/0260107920925675
Furulund KM, Molkken-stvold K (2007) Increasing software effort estimation accuracy using experience data, estimation models and checklists. In: Seventh international conference on quality software (QSIC 2007), IEEE, Portland, pp 342–347. https://doi.org/10.1109/QSIC.2007.4385518
Fægri TE (2010) Adoption of team estimation in a specialist organizational environment. In: Sillitti A, Martin A, Wang X, Whitworth E (eds) Agile processes in software engineering and extreme programming, Lecture Notes in Business Information Processing, Springer, Berlin, pp 28–42. https://doi.org/10.1007/978-3-642-13054-0_3
Gale NK, Heath G, Cameron E, Rashid S, Redwood S (2013) Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 13(1):117. https://doi.org/10.1186/1471-2288-13-117
Gandomani TJ, Koh TW, Binhamid AK (2014) A case study research on software cost estimation using experts’ estimates, Wideband Delphi, and Planning Poker technique. Int J Softw Eng Appl 8(11):173–182. https://doi.org/10.14257/ijseia.2014.8.11.16
Gandomani TJ, Faraji H, Radnejad M (2019) Planning poker in cost estimation in agile methods: averaging vs. consensus. In: 2019 5th conference on knowledge based engineering and innovation (KBEI), IEEE, Tehran, pp 066–071. https://doi.org/10.1109/KBEI.2019.8734960
Glass RL, Rost J, Matook MS (2008) Lying on software projects. IEEE Softw 25(6):90–95. https://doi.org/10.1109/MS.2008.150
Glenton C, Carlsen B, Lewin S, Munthe-Kaas H, Colvin CJ, Tunccalp O, Bohren MA, Noyes J, Booth A, Garside R, Rashidian A, Flottorp S, Wainwright M (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 5: how to assess adequacy of data. Implement Sci 13(1):14. https://doi.org/10.1186/s13012-017-0692-7
Grapenthin S, Book M, Richter T, Gruhn V (2016) Supporting feature estimation with risk and effort annotations. In: 2016 42th euromicro conference on software engineering and advanced applications (SEAA), IEEE, Limassol, pp 17–24. https://doi.org/10.1109/SEAA.2016.24
Grimstad S, Jørgensen M (2007) Inconsistency of expert judgment-based estimates of software development effort. J Syst Softw 80(11):1770–1777. https://doi.org/10.1016/j.jss.2007.03.001, . www.sciencedirect.com/science/article/pii/S0164121207000714
Grimstad S, Jørgensen M (2008) A preliminary study of sequence effects in judgment-based software development work-effort estimation. In: Proceedings of the 12th international conference on evaluation and assessment in software engineering, EASE’08, BCS Learning & Development Ltd., Swindon, pp 129–135. http://dl.acm.org/citation.cfm?id=2227115.2227129
Grimstad S, Jørgensen M, (2009) Preliminary study of sequence effects in judgment-based software development work-effort estimation. IET Softw 3(5):435–441. https://doi.org/10.1049/iet-sen.2008.0110
Grimstad S, Jorgensen M, Molokken-Ostvold K (2005) The clients’ impact on effort estimation accuracy in software development projects. In: 11th IEEE international software metrics symposium (METRICS’05), IEEE, Como, pp. 10. https://doi.org/10.1109/METRICS.2005.30
Guyatt GH, Oxman AD, Schünemann HJ, Tugwell P, Knottnerus A (2011) GRADE guidelines: a new series of articles in the journal of clinical epidemiology. J Clin Epidemiol 64(4):380–382. https://doi.org/10.1016/j.jclinepi.2010.09.011, . www.sciencedirect.com/science/article/pii/S089543561000329X
Halkjelsvik T, Jorgensen M (2011) To read two pages, I need 5 minutes, but give me 5 minutes and I will read four: how to change productivity estimates by inverting the question. Appl Cogn Psychol 25(2):314–323. https://doi.org/10.1002/acp.1693
Halkjelsvik T, Jørgensen M (2018a) How we predict time usage. In: Halkjelsvik T, Jørgensen M (eds) Time predictions: understanding and avoiding unrealism in project planning and everyday life, simula SpringerBriefs on computing, Springer International Publishing, Cham, pp 5–11, https://doi.org/10.1007/978-3-319-74953-2_2
Halkjelsvik T, Jørgensen M (2018) Time predictions: understanding and avoiding unrealism in project planning and everyday life. Simula SpringerBriefs on Computing, Springer International Publishing, Cham,. https://doi.org/10.1007/978-3-319-74953-2, www.springer.com/gp/book/9783319749525
Halkjelsvik T, Jørgensen M (2018c) Uncertainty of time predictions. In: Halkjelsvik T, Jørgensen M (eds) Time predictions: understanding and avoiding unrealism in project planning and everyday life, simula SpringerBriefs on cmputing, Springer International Publishing, Cham, pp 71–79. https://doi.org/10.1007/978-3-319-74953-2_5
Haran U, Ritov I, Mellers BA (2013) The role of actively open-minded thinking in information acquisition, accuracy, and calibration. Judgment Decision Making 8(3):188–201. https://ideas.repec.org/a/jdm/journl/v8y2013i3p188-201.html
Haugen NC (2006) An empirical study of using planning poker for user story estimation. In: AGILE 2006 (AGILE’06), IEEE, Minneapolis, pp 9–34, https://doi.org/10.1109/AGILE.2006.16
He M, Zhang H, Yang Y, Wang Q, Li M (2010) Understanding the influential factors to development effort in Chinese software industry. In: Ali Babar M, Vierimaa M, Oivo M (eds) Product-focused software process improvement, Lecture Notes in Computer Science, Springer, Berlin, pp 306–320. https://doi.org/10.1007/978-3-642-13792-1_24
Hoda R, Murugesan LK (2016) Multi-level agile project management challenges: a self-organizing team perspective. J Syst Softw 117:245–257. https://doi.org/10.1016/j.jss.2016.02.049, www.sciencedirect.com/science/article/pii/S0164121216000807
Hughes RT (1996) Expert judgement as an estimating method. Inf Softw Technol 38(2):67–75. https://doi.org/10.1016/0950-5849(95)01045-9, www.sciencedirect.com/science/article/pii/0950584995010459
Jorgensen M (2014) What we do and don’t know about software development effort estimation. IEEE Softw 31(2):37–40. https://doi.org/10.1109/MS.2014.49, https://ieeexplore.ieee.org/document/6774376/
Jorgensen M, Carelius GJ (2004) An empirical study of software project bidding. IEEE Trans Softw Eng 30(12):953–969. https://doi.org/10.1109/TSE.2004.92
Jorgensen M, Grimstad S (2012) Software development estimation biases: the role of interdependence. IEEE Trans Softw Eng 38(3):677–693. https://doi.org/10.1109/TSE.2011.40
Jorgensen M, Molokken-Ostvold K (2004) Reasons for software effort estimation error: impact of respondent role, information collection approach, and data analysis method. IEEE Trans Softw Eng 30(12):993–1007. https://doi.org/10.1109/TSE.2004.103, https://ieeexplore.ieee.org/document/1377193/
Jørgensen M (2011) Contrasting ideal and realistic conditions as a means to improve judgment-based software development effort estimation. Inf Softw Technol 53(12):1382–1390. https://doi.org/10.1016/j.infsof.2011.07.001
Jørgensen M (2013) Relative estimation of software development effort: it matters with what and how you compare. IEEE Softw 30(2):74–79. https://doi.org/10.1109/MS.2012.70
Jørgensen M (2015) The effect of the time unit on software development effort estimates. In: 2015 9th international conference on software, knowledge, information management and applications (SKIMA), IEEE, Kathmandu, Nepal, pp 1–5. https://doi.org/10.1109/SKIMA.2015.7399992
Jørgensen M (2016) Unit effects in software project effort estimation: work-hours gives lower effort estimates than workdays. J Syst Softw 117:274–281. https://doi.org/10.1016/j.jss.2016.03.048, www.sciencedirect.com/science/article/pii/S0164121216300085
Jørgensen M, Escott E (2022) Relative estimates of software development effort: are they more accurate or less time-consuming to produce than absolute estimates, and to what extent are they person-independent? Inf Softw Technol 143. https://doi.org/10.1016/j.infsof.2021.106782, www.sciencedirect.com/science/article/pii/S0950584921002251
Jørgensen M, Grimstad S (2011) The impact of irrelevant and misleading information on software development effort estimates: a randomized controlled field experiment. IEEE Trans Softw Eng 37(5):695–707. https://doi.org/10.1109/TSE.2010.78
Jørgensen M, Gruschke TM (2009) The impact of lessons-learned sessions on effort estimation and uncertainty assessments. IEEE Trans Softw Eng 35(3):368–383. https://doi.org/10.1109/TSE.2009.2
Jørgensen M, Halkjelsvik T (2010) The effects of request formats on judgment-based effort estimation. J Syst Softw 83(1):29–36. https://doi.org/10.1016/j.jss.2009.03.076, www.sciencedirect.com/science/article/pii/S0164121209000879
Jørgensen M, Halkjelsvik T (2020) Sequence effects in the estimation of software development effort. J Syst Softw 159. https://doi.org/10.1016/j.jss.2019.110448, www.sciencedirect.com/science/article/pii/S0164121219302225
Jørgensen M, Sjøberg DIK (2001) Impact of effort estimates on software project work. Inf Softw Technol 43:10
Jørgensen M, Sjøberg DIK (2004) The impact of customer expectation on software development effort estimates. Int J Project Manag 22(4):317–325. https://doi.org/10.1016/S0263-7863(03)00085-1, www.sciencedirect.com/science/article/pii/S0263786303000851
Jørgensen M, Faugli B, Gruschke T (2007) Characteristics of software engineers with optimistic predictions. J Syst Softw 80(9):1472–1482. https://doi.org/10.1016/j.jss.2006.09.047, www.sciencedirect.com/science/article/pii/S0164121206002986
Jørgensen M, Boehm B, Rifkin S (2009) Software development effort estimation: formal models or expert judgment? IEEE Softw 26(2):14–19. https://doi.org/10.1109/MS.2009.47
Kahneman D, Rosenfield AM, Gandhi L, Blaser T (2016) Noise: how to overcome the high, hidden cost of inconsistent decision making. Harvard Business Rev October:36–43. https://hbr.org/2016/10/noise
Kahneman D, Sibony O, Sunstein CR (2021) Noise: a flaw in human judgment, vol 1, 1st edn. Little, Brown Spark, New York
Karna H, Gotovac S (2014) Estimators characteristics and effort estimation of software projects. 2014 9th international conference on software engineering and applications (ICSOFT-EA). IEEE, Vienna, Austria, pp 26–35
Keaveney S, Conboy K (2006) Cost estimation in agile development projects. In: ECIS 2006 Proceedings, AIS Library, Göteborg p 16
Kitchenham B, Linkman S (1997) Estimates, uncertainty, and risk. IEEE Softw 14(3):69–74. https://doi.org/10.1109/52.589239
Lagerström R, von Würtemberg LM, Holm H, Luczak O (2012) Identifying factors affecting software development cost and productivity. Softw Qual J 20(2):395–417. https://doi.org/10.1007/s11219-011-9137-8
Layman L, Nagappan N, Guckenheimer S, Beehler J, Begel A (2008) Mining software effort data: preliminary analysis of visual studio team system data. In: Proceedings of the 2008 international working conference on Mining software repositories, MSR’08, Association for Computing Machinery, New York, pp 43–46. https://doi.org/10.1145/1370750.1370762
Lederer A, Mirani R (1990) Information system cost estimating: a management perspective. Manag Inf Syst Quart 14(2):159–176, https://aisel.aisnet.org/misq/vol14/iss2/3
Lederer AL, Prasad J (1991) The validation of a political model of information systems development cost estimating. ACM SIGCPR Comput Person 13(2):47–57. https://doi.org/10.1145/122393.122398
Lederer AL, Prasad J (1995) Causes of inaccurate software development cost estimates. J Syst Softw 31(2):125–134. https://doi.org/10.1016/0164-1212(94)00092-2, http://linkinghub.elsevier.com/retrieve/pii/0164121294000922
Lee M, Rothenberger M, Peffers K (2011) Identifying effort estimation factors for corrective maintenance in object-oriented systems. In: AMCIS 2011 Proceedings, p 186. https://aisel.aisnet.org/amcis2011_submissions/186
Lenberg P, Feldt R, Wallgren LG (2014) Towards a behavioral software engineering. In: Proceedings of the 7th international workshop on cooperative and human aspects of software engineering, CHASE 2014, Association for Computing Machinery, New York, pp 48–55. https://doi.org/10.1145/2593702.2593711
Lenberg P, Feldt R, Wallgren LG (2015) Behavioral software engineering: a definition and systematic literature review. J Syst Softw 107:15–37. https://doi.org/10.1016/j.jss.2015.04.084
Lewin S, Bohren M, Rashidian A, Munthe-Kaas H, Glenton C, Colvin CJ, Garside R, Noyes J, Booth A, Tunçalp O, Wainwright M, Flottorp S, Tucker JD, Carlsen B (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table. Implement Sci 13(1):10. https://doi.org/10.1186/s13012-017-0689-2
Lewin S, Booth A, Glenton C, Munthe-Kaas H, Rashidian A, Wainwright M, Bohren MA, Tunçalp O, Colvin CJ, Garside R, Carlsen B, Langlois EV, Noyes J (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci 13(1):2. https://doi.org/10.1186/s13012-017-0688-3
Løhre E, Jørgensen M (2016) Numerical anchors and their strong effects on software development effort estimates. J Syst Softw 116:49–56. https://doi.org/10.1016/j.jss.2015.03.015, www.sciencedirect.com/science/article/pii/S0164121215000618
Magazinius A, Börjesson S, Feldt R (2012) Investigating intentional distortions in software cost estimation - an exploratory study. J Syst Softw 85(8):1770–1781. https://doi.org/10.1016/j.jss.2012.03.026
Magazinovic A, Pernstål J (2008) Any other cost estimation inhibitors? In: Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement - ESEM ’08, ACM Press, Kaiserslautern, p 233, https://doi.org/10.1145/1414004.1414042. http://portal.acm.org/citation.cfm?doid=1414004.1414042
Mahnič V, Hovelja T (2012) On using planning poker for estimating user stories. J Syst Softw 85(9):2086–2095. https://doi.org/10.1016/j.jss.2012.04.005, www.sciencedirect.com/science/article/pii/S0164121212001021
Makridakis S, Hyndman RJ, Petropoulos F (2020) Forecasting in social settings: the state of the art. Int J Forecast 36(1):15–28. https://doi.org/10.1016/j.ijforecast.2019.05.011. https://www.sciencedirect.com/science/article/pii/S0169207019301876
Mann A (2016) The power of prediction markets. Nature 538(7625):308–310. https://doi.org/10.1038/538308a, www.nature.com/articles/538308a
Matos O, Fortaleza L, Conte T, Mendes E (2013) Realising web effort estimation. In: Proceedings of the 17th international conference on evaluation and assessment in software engineering, Association for Computing Machinery, Porto de Galinhas, pp 12–23. https://doi.org/10.1145/2460999.2461002
Matsubara P, Gadelha B, Steinmacher I, Conte T (2021). Supplementary material for the SEXTAMT. https://doi.org/10.6084/m9.figshare.14502405.v2
Matsubara P, Steinmacher I, Gadelha B, Conte T (2021b) Buying time in software development: how estimates become commitments? In: Proceedings of the 14th international conference on cooperative and human aspects of software engineering, IEEE, Madrid, pp 61–70
Matsubara P, Gadelha B, Steinmacher I, Conte T (2022) SEXTAMT: a systematic map to navigate the wide seas of factors affecting expert judgment software estimates. J Syst Softw 185:111148. https://doi.org/10.1016/j.jss.2021.111148
Matsubara P, Gadelha B, Steinmacher I, Conte T (2023) Material for much more than a prediction. https://doi.org/10.6084/m9.figshare.19406945.v1
Mellers B, Stone E, Atanasov P, Rohrbaugh N, Metz SE, Ungar L, Bishop MM, Horowitz M, Merkle E, Tetlock P (2015) The psychology of intelligence analysis: drivers of prediction accuracy in world politics. J Exp Psychol Appl 21(1):1–14. https://doi.org/10.1037/xap0000040
Mellers B, Stone E, Murray T, Minster A, Rohrbaugh N, Bishop M, Chen E, Baker J, Hou Y, Horowitz M, Ungar L, Tetlock P (2015) Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspectives on Psychological Science 10(3):267–281. https://doi.org/10.1177/1745691615577794
Mendes FF, Mendes E, Salleh N (2019) The relationship between personality and decision-making: a systematic literature review. Inf Softw Technol 111:50–71. https://doi.org/10.1016/j.infsof.2019.03.010, www.sciencedirect.com/science/article/pii/S0950584919300576
Merriam-Webster (2021) Forecast. In Merriam-Webster.com dictionary. https://www.merriam-webster.com/dictionary/forecast
Mohanani R, Salman I, Turhan B, Rodríguez P, Ralph P (2020) Cognitive biases in software engineering: a systematic mapping study. IEEE Trans Softw Eng 46(12):1318–1339. https://doi.org/10.1109/TSE.2018.2877759
MoløkkenØstvold K, Jørgensen M (2004) Group processes in software effort estimation. Emp Softw Eng 9(4):315–334. https://doi.org/10.1023/B:EMSE.0000039882.39206.5a
Moløkken-Østvold K, Haugen NC, Benestad HC (2008) Using planning poker for combining expert estimates in software projects. J Syst Softw 81(12):2106–2117. https://doi.org/10.1016/j.jss.2008.03.058, www.sciencedirect.com/science/article/pii/S0164121208000885
Munthe-Kaas H, Bohren MA, Glenton C, Lewin S, Noyes J, Tunçalp O, Booth A, Garside R, Colvin CJ, Wainwright M, Rashidian A, Flottorp S, Carlsen B (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 3: how to assess methodological limitations. Implementation Sci 13(1):9. https://doi.org/10.1186/s13012-017-0690-9
Münscher R, Vetter M, Scheuerle T (2016) A review and taxonomy of choice architecture techniques. J Behav Dec Making 29(5):511–524. https://doi.org/10.1002/bdm.1897, http://onlinelibrary.wiley.com/doi/abs/10.1002/bdm.1897
Nowell LS, Norris JM, White DE, Moules NJ (2017) Thematic analysis: striving to meet the trustworthiness criteria. Int J Qual Methods 16(1):1609406917733847. https://doi.org/10.1177/1609406917733847
Noyes J, Booth A, Lewin S, Carlsen B, Glenton C, Colvin CJ, Garside R, Bohren MA, Rashidian A, Wainwright M, Tunçalp O, Chandler J, Flottorp S, Pantoja T, Tucker JD, Munthe-Kaas H (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 6: how to assess relevance of the data. Implement Sci 13(1):4. https://doi.org/10.1186/s13012-017-0693-6
Organization C (2021) Measurement Manual v5.0 Part 1 Principles, Defs Rules Tech Rep, COSMIC Organization, https://cosmic-sizing.org/publications/measurement-manual-v5-0-may-2020-part-1-principles-definitions-rules/
Petropoulos F, Apiletti D, Assimakopoulos V, Babai MZ, Barrow DK, Ben Taieb S, Bergmeir C, Bessa RJ, Bijak J, Boylan JE, Browell J, Carnevale C, Castle JL, Cirillo P, Clements MP, Cordeiro C, Cyrino Oliveira FL, De Baets S, Dokumentov A, Ellison J, Fiszeder P, Franses PH, Frazier DT, Gilliland M, Gönül MS, Goodwin P, Grossi L, Grushka-Cockayne Y, Guidolin M, Guidolin M, Gunter U, Guo X, Guseo R, Harvey N, Hendry DF, Hollyman R, Januschowski T, Jeon J, Jose VRR, Kang Y, Koehler AB, Kolassa S, Kourentzes N, Leva S, Li F, Litsiou K, Makridakis S, Martin GM, Martinez AB, Meeran S, Modis T, Nikolopoulos K, Önkal D, Paccagnini A, Panagiotelis A, Panapakidis I, Pavía JM, Pedio M, Pedregal DJ, Pinson P, Ramos P, Rapach DE, Reade JJ, Rostami-Tabar B, Rubaszek M, Sermpinis G, Shang HL, Spiliotis E, Syntetos AA, Talagala PD, Talagala TS, Tashman L, Thomakos D, Thorarinsdottir T, Todini E, Trapero Arenas JR, Wang X, Winkler RL, Yusupova A, Ziel F (2022) Forecasting: theory and practice. International Journal of Forecasting In press. https://doi.org/10.1016/j.ijforecast.2021.11.001, www.sciencedirect.com/science/article/pii/S0169207021001758
Rahikkala J, Hyrynsalmi S, Leppänen V (2015) Accounting testing in software cost estimation: a case study of the current practice and impacts. In: 14th symposium on programming languages and software tools, Tampere, Finland, p 15
Rahikkala J, Hyrynsalmi S, Leppänen V, Porres I (2018) The role of organisational phenomena in software cost estimation: a case study of supporting and Hindering factors. e-Inform Softw Eng J 12(1):167–198. https://doi.org/10.5277/e-Inf180107. http://www.e-informatyka.pl/attach/e-Informatica_-_Volume_12/eInformatica2018Art7.pdf
Satopää VA, Salikhov M, Tetlock PE, Mellers B (2021) Bias, information, noise: the BIN model of forecasting. Manag Sci 1–20. https://doi.org/10.1287/mnsc.2020.3882, http://pubsonline.informs.org/doi/10.1287/mnsc.2020.3882
Schneider WJ, McGrew KS (2018) The Cattell-Horn-Carroll theory of cognitive abilities. In: Contemporary intellectual assessment: theories, tests, and issues, 4th ed, 4th edn, The Guilford Press, New York, pp 73–163
Shepperd M, Mair C, Jørgensen M (2018) An experimental evaluation of a de-biasing intervention for professional software developers. In: Proceedings of the 33rd annual acm symposium on applied computing- SAC ’18, ACM, New York, pp 1510–1517. https://doi.org/10.1145/3167132.3167293
Simon HA (2000) Bounded rationality in social science: today and tomorrow. Mind Soc 1(1):25–39. https://doi.org/10.1007/BF02512227
Svedholm-Häkkinen AM, Lindeman M (2018) Actively open-minded thinking: development of a shortened scale and disentangling attitudes towards knowledge and people. Think Reason 24(1):21–40. https://doi.org/10.1080/13546783.2017.1378723
Tamrakar R, Jørgensen M (2012) Does the use of Fibonacci numbers in planning poker affect effort estimates? In: 16th international conference on evaluation assessment in software engineering (EASE 2012), IET, Ciudad Real, pp 228–232. https://doi.org/10.1049/ic.2012.0030
Thaler RH (2018) From cashews to nudges: the evolution of behavioral economics. Am Econ Rev 108(6):1265–1287. https://doi.org/10.1257/aer.108.6.1265, www.aeaweb.org/articles?id=10.1257/aer.108.6.1265
Thaler RH, Sunstein CR (2021) Nudge: the final edition, final, edition. Penguin Books, New York
Timon CE (2020) Defining the new behavioral science(s). Signs Soc 8(3):472–496. https://doi.org/10.1086/710840, www.journals.uchicago.edu/doi/full/10.1086/710840
Trendowicz A, Münch J, Jeffery R (2011) State of the practice in software effort estimation: a survey and literature review. In: Huzar Z, Koci R, Meyer B, Walter B, Zendulka J (eds) Software engineering techniques - Lecture Notes in Computer Science. Springer, Berlin Heidelberg, Berlin, pp 232–245
Tversky A, Kahneman D (1973) Availability: a heuristic for judging frequency and probability. Cognitive Psychology 5(2):207–232. https://doi.org/10.1016/0010-0285(73)90033-9, www.sciencedirect.com/science/article/pii/0010028573900339
Tversky A, Kahneman D (1974) Judgment under uncertainty: heuristics and biases. Science 185(4157):1124–1131. https://doi.org/10.1126/science.185.4157.1124
Usman M, Mendes E, Börstler J (2015) Effort estimation in agile software development: a survey on the state of the practice. In: Proceedings of the 19th international conference on evaluation and assessment in software engineering, EASE ’15, Association for Computing Machinery, Nanjing, pp 1–10. https://doi.org/10.1145/2745802.2745813
Usman M, Börstler J, Petersen K (2017) An effort estimation taxonomy for agile software development. Int J Softw Eng Knowl Eng 27(04):641–674. https://doi.org/10.1142/S0218194017500243, www.worldscientific.com/doi/10.1142/S0218194017500243
Usman M, Britto R, Damm LO, Börstler J (2018) Effort estimation in large-scale software development: an industrial case study. Inf Softw Technol 99:21–40. https://doi.org/10.1016/j.infsof.2018.02.009, www.sciencedirect.com/science/article/pii/S0950584918300326
Usman M, Petersen K, Börstler J, Santos Neto P (2018) Developing and using checklists to improve software effort estimation: a multi-case study. J Syst Softw 146:286–309. https://doi.org/10.1016/j.jss.2018.09.054, www.sciencedirect.com/science/article/pii/S0164121218302073
VandenBos G (2015) Bandwagon effect. https://doi.org/10.1037/14646-000
Wohlin C, Rainer A (2021) Challenges and recommendations to publishing and using credible evidence in software engineering. Inf Softw Technol 134. https://doi.org/10.1016/j.infsof.2021.106555, www.sciencedirect.com/science/article/pii/S0950584921000409
Yamagishi T, Li Y, Takagishi H, Matsumoto Y, Kiyonari T (2014) In search of Homo economicus. Psychol Sci 25(9):1699–1711. https://doi.org/10.1177/0956797614538065
Yang D, Wang Q, Li M, Yang Y, Ye K, Du J (2008) A survey on software cost estimation in the chinese software industry. In: Proceedings of the second ACM-IEEE international symposium on empirical software engineering and measurement - ESEM ’08, ACM Press, Kaiserslautern, p 253. https://doi.org/10.1145/1414004.1414045. http://portal.acm.org/citation.cfm?doid=1414004.1414045
Zapata AH, Chaudron MRV (2013) An empirical study into the accuracy of it estimations and its influencing factors. Int J Softw Eng Knowl Eng 23(04):409–432. https://doi.org/10.1142/S0218194013400081, www.worldscientific.com/doi/abs/10.1142/S0218194013400081
Zarour A, Zein S (2019) Software development estimation techniques in industrial contexts: an exploratory multiple case-study. Int J Technol Educ Sci 3(2):72–84. https://eric.ed.gov/?id=EJ1227141
Łabȩdzki M, Promiński P, Rybicki A, Wolski M (2017) Agile effort estimation in software development projects - case study. Central Eur Rev Econ Manag 1(3). https://doi.org/10.29015/cerem.359
Funding
We thank the reviewers for all their suggestions, many of which we incorporated into the paper and significantly improved it. The present work is the result of the Research and Development (R &D) project 001/2020, signed with the Federal University of Amazonas and FAEPI, Brazil, which has funding from Samsung, using resources from the Informatics Law for the Western Amazon (Federal Law nº 8.387/1991), and its disclosure is in accordance with article 39 of Decree No. 10.521/2020. Also supported by the Federal University of Mato Grosso do Sul (UFMS), the Federal University of Amazonas (UFAM), CAPES - Financing Code 001, CNPq processes 314174/2020-6 and 313067/2020-1, and FAPEAM process 062.00150/2020, and grant #2020/05191-2 São Paulo Research Foundation (FAPESP).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare that they have no conflict of interest.
Additional information
Communicated by: Burak Turhan.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Matsubara, P.G.F., Steinmacher, I., Gadelha, B. et al. Much more than a prediction: Expert-based software effort estimation as a behavioral act. Empir Software Eng 28, 98 (2023). https://doi.org/10.1007/s10664-023-10332-9
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-023-10332-9