Hans Christian Benestad
ABSTRACT
Context: To select the essential, non-negotiable product features is a key skill for stakeholders in software projects. Such selection relies on human judgment, possibly supported by structured prioritization techniques and tools. Goal: Our goal was to investigate whether certain attributes of prioritization techniques affect stakeholders' threshold for judging product features as essential. The four investigated techniques represent four combinations of granularity (low, high) and cognitive support (low, high). Method: To control for robustness and masking effects when investigating in the field, we conducted both an artificial experiment and a field experiment using the same prioritization techniques. In the artificial experiment, 94 subjects in four treatment groups indicated the features (from a list of 16) essential when buying a new cell phone. In the field experiment, 44 domain experts indicated the software product features that were essential for the fulfillment of the project's vision. The effects of granularity and cognitive support on the number of essential ratings were analyzed and compared between the experiments. Result: With lower granularity, significantly more features were rated as essential. The effect was large in the general experiment and extreme in the field experiment. Added cognitive support had medium effect, but worked in opposite directions in the two experiments, and was not statistically significant in the field experiment. Implications: Software projects should avoid taking stakeholders' judgments of essentiality at face value. Practices and tools should be designed to counteract biases and to support the conscious knowledge-based elements of prioritizing.
- J. S. Armstrong, editor. Principles of Forecasting: A Handbook for Researchers and Practitioners. Kluwer Academic Publishers, 2001.Google Scholar
Cross Ref
- H. C. Benestad and J. E. Hannay. A comparison of model-based and judgment-based release planning in incremental software projects. In Proc. 33rd Int'l Conf. Software Engineering (ICSE 2011), pages 766--775. ACM, 2011. Google ScholarDigital Library
- P. Berander and A. Andrews. Requirements prioritization. In Engineering and Managing Software Requirements, chapter 4, pages 69--94. Springer, 2005.Google ScholarCross Ref
- P. Berander and P. Jönsson. Hierarchical cumulative voting (hcv) prioritization of requirements in hierarchies. Int'l J. Software Engineering & Knowledge Engineering, 16:819--849, 2006.Google ScholarCross Ref
- P. Berander, K. A. Khan, and L. Lehtola. Towards a research framework on requirements prioritization. In Proc. 6th Conf. Software Engineering Research and Practice in Sweden, pages 39--48, 2006.Google Scholar
- A. S. Danesh and R. Ahmad. Study of prioritization techniques using students as subjects. In Int'l Conf. Information Management and Engineering, pages 390--394. IEEE Computer Society, 2009. Google ScholarDigital Library
- K. A. Ericsson. The influence of experience and deliberate practice on the development of superior expert performance. In K. A. Ericsson, N. Charness, P. J. Feltovich, and R. R. Hoffman, editors, The Cambridge Handbook of Expertise and Expert Performance, chapter 38, pages 683--703. Cambridge Univ. Press, 2006.Google Scholar
- D. Gentner and A. L. Stevens, editors. Mental Models. Lawrence Erlbaum Associates, Inc., 1983.Google Scholar
- G. A. Gescheider. Psychophysical scaling. Annual Review of Psychology, 39:169--200, 1988.Google ScholarCross Ref
- G. Gigerenzer and P. M. Todd, editors. Simple Heuristics that Make Us Smart. Oxford University Press, 1999.Google Scholar
- T. Halkjelsvik and M. Jørgensen. From origami to software development: A review of studies on judgment-based predictions of performance time. accepted to Psychological Bulletin, 2011.Google Scholar
- J. E. Hannay. Better software effort estimationâa matter of skill or environment? Submitted to IEEE Trans. Software Engineering; available at simula.no/people/johannay/bibliography, 2012.Google Scholar
- J. E. Hannay and H. C. Benestad. Perceived productivity threats in large agile development projects. In Proc. 4th Int'l Symp.Empirical Software Engineering and Measurement (ESEM), pages 1--10. IEEE Computer Society, 2010. Google ScholarDigital Library
- J. E. Hannay and M. Jørgensen. The role of deliberate artificial design elements in software engineering experiments. IEEE Trans. Software Eng., 34:242--259, Mar/Apr 2008. Google ScholarDigital Library
- P. T. Harker. Incomplete pairwise comparisons in the analytic hierarchy process. Mathematical Modelling, 9(11):837--848, 1987.Google ScholarCross Ref
- P. N. Johnson-Laird. Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Cambridge Univ. Press, 1983. Google ScholarDigital Library
- M. Jørgensen and S. Grimstad. The impact of irrelevant and misleading information on software development effort estimates: A randomized controlled field experiment. IEEE Trans. Software Eng., 37(5):695--707, 2011. Google ScholarDigital Library
- M. Jørgensen and T. Halkjelsvik. The effects of request formats on judgment-based effort estimation. J. Systems and Software, 83(1):29--36, 2010. Google ScholarDigital Library
- D. Kahneman and S. Frederick. A model of heuristic judgment. In K. J. Holyoak and R. G. Morrison, editors, The Cambridge Handbook of Thinking and Reasoning, pages 267--294. Cambridge Univ. Press, 2004.Google Scholar
- V. B. Kampenes, T. Dybå, J. E. Hannay, and D. I. K. Sjøberg. A systematic review of effect size in software engineering experiments. Information and Software Technology, 49(11-12):1073--1086, Nov. 2007. Google ScholarDigital Library
- J. Karlsson. Software requirements prioritizing. In 2nd Int'l Conf. Requirements Engineering (ICRE'96), pages 110--116. IEEE Computer Society, 1996. Google ScholarDigital Library
- J. Karlsson, S. Olsson, and K. Ryan. Improved practical support for large-scale requirements prioritising. Requirements Engineering, 2:51--60, 1997.Google ScholarDigital Library
- J. Karlsson, C. Wohlin, and B. Regnell. An evaluation of methods for prioritizing software requirements. Information & Software Technology, 39(14-15):939--947, 1998.Google ScholarCross Ref
- L. Karlsson, M. Höst, and B. Regnell. Evaluating the practical use of different measurement scales in requirements prioritisation. In Proc. 2006 ACM/IEEE Int'l Symp. Empirical Software Engineering, ISESE '06, pages 326--335. ACM, 2006. Google ScholarDigital Library
- L. Karlsson, T. Thelin, B. Regnell, P. Berander, and C. Wohlin. Pair-wise comparisons versus planning game partitioning-experiments on requirements prioritisation techniques. Empirical Software Engineering, 12:3--33, 2007. Google ScholarDigital Library
- G. Klein. Developing expertise in decision making. Thinking & Reasoning, 3(4):337--352, 1997.Google ScholarCross Ref
- L. Lehtola and M. Kauppinen. Empirical evaluation of two requirements prioritization methods in product development projects. In T. Dingsøyr, editor, Software Process Improvement, volume 3281 of Lecture Notes in Computer Science, pages 161--170. Springer, 2004.Google Scholar
- L. Lehtola and M. Kauppinen. Suitability of requirements prioritization methods for market-driven software product development. Software Process: Improvement and Practice, 11:7--19, 2006.Google ScholarCross Ref
- T. Mussweiler. Comparison processes in social judgment: Mechanisms and consequences. Psych. Review, 110(3):472--489, 2003.Google ScholarCross Ref
- A. Parducci and D. H. Wedell. The category effect with rating scales: Number of categories, number of stimuli, and method of presentation. J. Experimental Psychology: Human Perception and Performance, 12(4):496--516, 1996.Google ScholarCross Ref
- A. Perini, F. Ricca, and A. Susi. Tool-supported requirements prioritization: Comparing the ahp and cbrank methods. Information and Software Technology, 51(6):1021--1032, 2009. Google ScholarDigital Library
- E. C. Poulton. Behavioral Decision Theory: A New Approach. Cambridge University Press, 1994.Google Scholar
- T. L. Saaty. The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation. McGraw-Hill, 1990.Google Scholar
- T. L. Saaty. Multicriteria Decision Making: The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation. RWS Publications, 1990.Google Scholar
- W. R. Shadish, T. D. Cook, and D. T. Campbell. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin, 2002.Google Scholar
- H. A. Simon. The Sciences of the Artificial. MIT Press, third edition, 1996. Google ScholarDigital Library
- F. Strack and T. Mussweiler. Explaining the enigmatic anchoring effect: Mechanisms of selective accessibility. J. Personality and Social Psychology, 73(3):437--446, 1997.Google ScholarCross Ref
- R. K. Yin. Case Study Research: Design and Methods, volume 5 of Applied Social Research Methods Series. Sage Publications, third edition, 2003.Google Scholar
Index Terms
- Does the prioritization technique affect stakeholders' selection of essential software product features?
Recommendations
A systematic literature review of software requirements prioritization research
Context: During requirements engineering, prioritization is performed to grade or rank requirements in their order of importance and subsequent implementation releases. It is a major step taken in making crucial decisions so as to increase the economic ...
A questionnaire-based survey methodology for systematically validating goal-oriented models
Goal models represent interests, intentions, and strategies of different stakeholders. Reasoning about the goals of a system unavoidably involves the transformation of unclear stakeholder requirements into goal-oriented models. The ability to validate ...
ReproTizer: A Fully Implemented Software Requirements Prioritization Tool
Transactions on Computational Collective Intelligence XXII - Volume 9655Before software is developed, requirements are elicited. These requirements could be over-blown or under-estimated in a way that meeting the expectations of stakeholders becomes a challenge. To develop a software that precisely meets the expectations of ...
Comments