Decision SupportHow stock-flow failure and general cognitive ability impact performance in operational dynamic control tasks
Introduction
Controlling dynamic operational systems is an important part of many jobs in all kinds of organizations. A production manager's responsibility to steer inventories or resources in a production system towards performance-maximizing levels is a classic example. We take a behavioral operational research perspective, as recently advocated by Hämäläinen et al., 2013, O'Keefe, 2016, and Brocklesby (2016), and aim to increase our understanding of individual factors’ impact on performance in these kinds of tasks. Previous research has provided evidence on a variety of factors, such as mental ability, cognitive reflection, knowledge and personality traits, in a broad range of domains, including inventory management (Croson and Donohue, 2006, Diehl and Sterman, 1995, Moritz et al., 2013, Strohhecker and Größler, 2013), supply chain management (Croson et al., 2014, Sterman, 1989b), capacity planning (Anderson et al., 2005, Strohhecker and Größler, 2012), and new product development (Paich and Sterman, 1993, van Oorschot et al., 2013). We follow Becker’s (2016, p. 807) call to “step out of the traditional paradigms” of operational research and adopt an industrial-organizational (IO) psychology perspective, which guides us to focus on human cognitive abilities (e.g., Carroll, 1993, McGrew, 2009, Spearman, 1927) while also considering personality factors and interests as control variables (e.g., Ackerman, 1996, Penney et al., 2011). Inspired by IO psychology structural theories of intelligence, which distinguish between general and specific cognitive abilities (Carroll, 1993, McGrew, 2009), we are especially interested in investigating individuals’ specific cognitive stock-flow-thinking ability (SFTA), also termed stock-flow failure (Cronin, Gonzalez, & Sterman, 2009), and distinguishing it from general cognitive ability (GCA).
SFTA is of high interest and relevance in the context of dynamic control tasks for two reasons. First, it has been uncovered as a potentially important factor by operational research on dynamic systems (Booth Sweeney and Sterman, 2000, Sterman and Sweeney, 2002) – that is, from within the operations research community. Second, since then, evidence has accumulated that human SFTA is poor: various studies have shown that surprisingly high percentages of even well-educated adults fail to solve very basic paper-and-pencil stock-and-flow tasks correctly (Booth Sweeney and Sterman, 2000, Cronin and Gonzalez, 2007, Cronin et al., 2009, Hämäläinen et al., 2013, Sterman, 2010, Sterman and Sweeney, 2007).
As dynamic operational systems, such as production systems, are built from the basic elements of stocks (accumulations over time, resulting in delays), flows (changing accumulations), and feedback loops (balancing, as well as reinforcing, causal cycles) (Coyle, 1983, Forrester, 1958, Größler et al., 2008), individual differences in SFTA are very likely a relevant factor. Managers with low SFTA should have difficulties in adequately controlling a system's flows so that the system's state approaches its target level resulting in poor operational control performance (P). Surprisingly, despite the abovementioned considerable research into the stock-flow failure phenomenon, empirical evidence on the relation between SFTA and performance is largely missing. In a pilot study, Strohhecker (2009) did not find significant bivariate correlations between SFTA and cumulated stock-out and inventory holding cost in a stochastic inventory management task. However, based on only 25 participants, this nonfinding lacks statistical power. Another nonfinding was reported by Strohhecker and Größler (2015), who did not find support for their hypothesis that participants who performed well in the tangible dynamic control task of filling a glass through a funnel also showed fewer stock-flow failures in a paper-and-pencil test, and vice versa. Providing empirical evidence on the SFTA-P link would therefore contribute to closing a relevant gap in this literature.
Given the IO psychology perspective that we have chosen, GCA is this study's obvious second focal individual factor for several reasons. First, it is a theoretically important explanatory factor (e.g., Ackerman, 1996, Furnham, 2008), and substantial evidence has shown that GCA is a major predictor of task and job performance, independent of the context (e.g., Oh, 2014, Scherbaum et al., 2012, Schmidt and Hunter, 1998). Second, the related research stream on complex problem solving has provided evidence on GCA's role in predicting performance (e.g., Beckmann and Guthke, 1995, Wittmann and Hattrup, 2004). Third, behavioral operations management research has shown that GCA significantly impacts decision-making outcomes in operations environments (e.g., Moritz et al., 2014, Strohhecker and Größler, 2013). Most of the studies mentioned above that have investigated the impact of individual factors on performance measures have used simple, linear explanatory models that include only one-step relations between independent and dependent variables, without (or with limited) consideration of moderating or mediating effects. Guided by IO psychology structural and investment theories of intelligence (Ackerman, 1996, Carroll, 1993, Cattell, 1987, McGrew, 2009), and by recently uncovered nonlinear effects of general mental ability on pay (Ganzach, Gotlibobski, Greenberg, & Pazy, 2013), we extend previous research in two important dimensions. First, we propose that SFTA is considered a specific cognitive ability, so that GCA invested in specific stock-flow-related experiences crystallizes in SFTA (Cattell, 1963, Cattell, 1987). As a consequence, we are not only interested in investigating direct effects, but also in revealing a potential mediating effect of GCA over SFTA on performance. Second, we advance previous research by not just adopting the well-established relation between GCA and performance, but also investigating a potentially nonlinear shape and thereby advancing our understanding of marginal effects.
Thus, we follow a growing stream of behavioral operational research – as encouraged, e.g., by Becker (2016), Hämäläinen et al., 2013, O'Keefe, 2016, and Brocklesby (2016) – and ask the following three research questions: First, does SFTA really relate to an individual decision maker's performance in controlling a dynamic operational system? Second, is SFTA a specific crystallized ability that is related to GCA, and, if so, what is the relative strength of the effects of SFTA and GCA, respectively, on performance, separating direct and indirect effects? Third, is GCA's direct effect on performance replicable and indeed nonlinear (or rather linear)? The analysis of both GCA and SFTA is not only theoretically important, as it adds to our understanding of the impact of individual differences on operational dynamic decision making, but is also of practical relevance to the field of operations research; for instance, a strong influence of SFTA on performance would call for operations research tools to be used to increase the ability of stock-flow thinking (as proposed by Sterman (2010), or to reduce this cognitive bias (as suggested by Hämäläinen et al. (2013). On the contrary, a strong influence of GCA on performance would prompt operations researchers to develop tools that moderate its impact on performance. In addition, investigating the shape of the relation between GCA and performance is practically relevant, as an improved understanding of this relation would, for instance, allow us to improve tools supporting personnel scheduling – an area that is frequently explored by the operations research community (Van den Bergh, Beliën, De Bruecker, Demeulemeester, & De Boeck, 2013). For example, the tradeoff between performance losses via use of personnel with lower abilities, and cost savings because of lower wages, could be managed.
We aim to answer the research questions by setting up an experiment in which participants are placed in a typical dynamic production and inventory control setting similar to that used by Strohhecker and Größler (2013). The participants have to make repeated decisions on production quantities of a nonperishable good that is put in inventory, and from there delivered to business customers. Following related behavioral research (e.g., Moritz et al., 2013, Narayanan and Moritz, 2015), performance is measured as an outcome orientation using cumulated profit and various cost measures as alternate proxies.
The remainder of this article is organized as follows. First, we elaborate on the theoretical background from which our hypotheses are derived. Second, we describe our method, including our experimental design, measures, participants, and procedure. Third, we present and discuss our results. Finally, we describe our study's limitations and future research avenues.
Section snippets
Operational control, performance, and SFTA
Größler et al. (2008) emphasized that accumulation processes, delays, and feedback loops are widespread phenomena in operational systems that have not always been considered appropriately by managers who dynamically control these systems. From a structural system dynamics perspective, these three phenomena emerge from stocks and flows as building blocks, which are connected by information links (Forrester, 1961, Sterman, 2000). Executing dynamic first-order control in such a system, as shown in
Experimental design and measures
An observational research design with five observations obtained in a laboratory setting, plus additional data retrieved from archival sources, was deemed appropriate to test our research model including H1 to H3.2, as shown in Fig. 2 (Trochim & Donnelly, 2007).
The dependent variable, performance, was measured following the well-established experimental paradigm for investigating dynamic decision making (Brehmer, 1992) using a computer-simulated micro world. The micro world was nearly identical
Results
Table 1 shows the descriptive statistics and correlations between all variables that we investigated. We find highly significant medium to strong bivariate correlations for GCA, SFTA, and P that do not differ much between parametric and nonparametric methods.
To test our hypotheses, we calculated the research model as proposed in Fig. 2. The results are depicted in Fig. 8.
Based on the results shown in Fig. 8, we find support for H1, suggesting that SFTA indeed directly influences performance.
Theoretical contributions
From a theoretical angle, our analysis targets the influence of individual factors on operational decision-making performance in a dynamic stock-flow system that is presented as a typical production control environment. By finding significant support for our mediating model (including concavity), we contribute to behavioral operational research in multiple ways.
First, we provide evidence regarding the specific cognitive ability, SFTA, showing that it is an important predictor of performance in
References (104)
A theory of adult intellectual development: Process, personality, interests, and knowledge
Intelligence
(1996)An outlook on behavioural OR – Three tasks, three pitfalls, one definition
European Journal of Operational Research
(2016)- et al.
Spearman's law of diminishing returns. A meta-analysis
Intelligence
(2017) Dynamic decision making: Human control of complex systems
Acta Psychologica
(1992)The what, the why and the how of behavioural operational research–An invitation to potential sceptics
European Journal of Operational Research
(2016)- et al.
A comparison of salient rewards in experiments: Money and class points
Economics Letters
(2001) The technical elements of the system dynamics approach
European Journal of Operational Research
(1983)Relations among general intelligence (g), aptitude tests, and GPA: Linear effects dominate
Intelligence
(2015)- et al.
Why don't well-educated adults understand accumulation? A challenge to researchers, educators, and citizens
Organizational Behavior & Human Decision Processes
(2009) - et al.
Effects of feedback complexity on dynamic decision making
Organizational Behavior and Human Decision Processes
(1995)