Decision Support
How stock-flow failure and general cognitive ability impact performance in operational dynamic control tasks

https://doi.org/10.1016/j.ejor.2019.01.048Get rights and content

Highlights

  • Behavioral study of individual factors predicting human control performance.

  • Focus on stock-flow thinking and general cognitive ability.

  • First to find robust support for a mediating causal model of cognitive abilities.

  • First to show that stock-flow thinking significantly affects control performance.

  • Practical implications for employee selection and training.

Abstract

Previous research has highlighted the importance of individual factors predicting human beings’ relatively poor performance in controlling dynamic operational systems. Taking an industrial and organizational (IO) psychology perspective within a behavioral operations research context, this study focuses on erroneous stock-flow thinking and general cognitive ability. It uses data from laboratory experiments to test a mediating structural equation model firmly grounded in IO psychology theory, proposing both a direct concave impact of general cognitive ability on control task performance and an indirect concave effect mediated by stock-flow thinking. With our results we are the first to demonstrate that stock-flow thinking serves as a mediator of general cognitive ability on performance, while at the same time, general cognitive ability non-linearly impacts performance. These findings contribute to theory building in behavioral operational research and demonstrate how adoption of perspectives other than the traditional paradigms used can help. It also has practical implications as it improves the relative assessment of different levers to increase operational control performance. We find that the total impact of general cognitive ability on control performance is higher than the direct effect of stock-flow thinking. Therefore, companies should add stock-flow thinking tests to their personnel selection toolbox, but put more weight on selecting operational managers with high cognitive abilities and somewhat less weight on training efforts that aim to reduce stock-flow failures.

Introduction

Controlling dynamic operational systems is an important part of many jobs in all kinds of organizations. A production manager's responsibility to steer inventories or resources in a production system towards performance-maximizing levels is a classic example. We take a behavioral operational research perspective, as recently advocated by Hämäläinen et al., 2013, O'Keefe, 2016, and Brocklesby (2016), and aim to increase our understanding of individual factors’ impact on performance in these kinds of tasks. Previous research has provided evidence on a variety of factors, such as mental ability, cognitive reflection, knowledge and personality traits, in a broad range of domains, including inventory management (Croson and Donohue, 2006, Diehl and Sterman, 1995, Moritz et al., 2013, Strohhecker and Größler, 2013), supply chain management (Croson et al., 2014, Sterman, 1989b), capacity planning (Anderson et al., 2005, Strohhecker and Größler, 2012), and new product development (Paich and Sterman, 1993, van Oorschot et al., 2013). We follow Becker’s (2016, p. 807) call to “step out of the traditional paradigms” of operational research and adopt an industrial-organizational (IO) psychology perspective, which guides us to focus on human cognitive abilities (e.g., Carroll, 1993, McGrew, 2009, Spearman, 1927) while also considering personality factors and interests as control variables (e.g., Ackerman, 1996, Penney et al., 2011). Inspired by IO psychology structural theories of intelligence, which distinguish between general and specific cognitive abilities (Carroll, 1993, McGrew, 2009), we are especially interested in investigating individuals’ specific cognitive stock-flow-thinking ability (SFTA), also termed stock-flow failure (Cronin, Gonzalez, & Sterman, 2009), and distinguishing it from general cognitive ability (GCA).

SFTA is of high interest and relevance in the context of dynamic control tasks for two reasons. First, it has been uncovered as a potentially important factor by operational research on dynamic systems (Booth Sweeney and Sterman, 2000, Sterman and Sweeney, 2002) – that is, from within the operations research community. Second, since then, evidence has accumulated that human SFTA is poor: various studies have shown that surprisingly high percentages of even well-educated adults fail to solve very basic paper-and-pencil stock-and-flow tasks correctly (Booth Sweeney and Sterman, 2000, Cronin and Gonzalez, 2007, Cronin et al., 2009, Hämäläinen et al., 2013, Sterman, 2010, Sterman and Sweeney, 2007).

As dynamic operational systems, such as production systems, are built from the basic elements of stocks (accumulations over time, resulting in delays), flows (changing accumulations), and feedback loops (balancing, as well as reinforcing, causal cycles) (Coyle, 1983, Forrester, 1958, Größler et al., 2008), individual differences in SFTA are very likely a relevant factor. Managers with low SFTA should have difficulties in adequately controlling a system's flows so that the system's state approaches its target level resulting in poor operational control performance (P). Surprisingly, despite the abovementioned considerable research into the stock-flow failure phenomenon, empirical evidence on the relation between SFTA and performance is largely missing. In a pilot study, Strohhecker (2009) did not find significant bivariate correlations between SFTA and cumulated stock-out and inventory holding cost in a stochastic inventory management task. However, based on only 25 participants, this nonfinding lacks statistical power. Another nonfinding was reported by Strohhecker and Größler (2015), who did not find support for their hypothesis that participants who performed well in the tangible dynamic control task of filling a glass through a funnel also showed fewer stock-flow failures in a paper-and-pencil test, and vice versa. Providing empirical evidence on the SFTA-P link would therefore contribute to closing a relevant gap in this literature.

Given the IO psychology perspective that we have chosen, GCA is this study's obvious second focal individual factor for several reasons. First, it is a theoretically important explanatory factor (e.g., Ackerman, 1996, Furnham, 2008), and substantial evidence has shown that GCA is a major predictor of task and job performance, independent of the context (e.g., Oh, 2014, Scherbaum et al., 2012, Schmidt and Hunter, 1998). Second, the related research stream on complex problem solving has provided evidence on GCA's role in predicting performance (e.g., Beckmann and Guthke, 1995, Wittmann and Hattrup, 2004). Third, behavioral operations management research has shown that GCA significantly impacts decision-making outcomes in operations environments (e.g., Moritz et al., 2014, Strohhecker and Größler, 2013). Most of the studies mentioned above that have investigated the impact of individual factors on performance measures have used simple, linear explanatory models that include only one-step relations between independent and dependent variables, without (or with limited) consideration of moderating or mediating effects. Guided by IO psychology structural and investment theories of intelligence (Ackerman, 1996, Carroll, 1993, Cattell, 1987, McGrew, 2009), and by recently uncovered nonlinear effects of general mental ability on pay (Ganzach, Gotlibobski, Greenberg, & Pazy, 2013), we extend previous research in two important dimensions. First, we propose that SFTA is considered a specific cognitive ability, so that GCA invested in specific stock-flow-related experiences crystallizes in SFTA (Cattell, 1963, Cattell, 1987). As a consequence, we are not only interested in investigating direct effects, but also in revealing a potential mediating effect of GCA over SFTA on performance. Second, we advance previous research by not just adopting the well-established relation between GCA and performance, but also investigating a potentially nonlinear shape and thereby advancing our understanding of marginal effects.

Thus, we follow a growing stream of behavioral operational research – as encouraged, e.g., by Becker (2016), Hämäläinen et al., 2013, O'Keefe, 2016, and Brocklesby (2016) – and ask the following three research questions: First, does SFTA really relate to an individual decision maker's performance in controlling a dynamic operational system? Second, is SFTA a specific crystallized ability that is related to GCA, and, if so, what is the relative strength of the effects of SFTA and GCA, respectively, on performance, separating direct and indirect effects? Third, is GCA's direct effect on performance replicable and indeed nonlinear (or rather linear)? The analysis of both GCA and SFTA is not only theoretically important, as it adds to our understanding of the impact of individual differences on operational dynamic decision making, but is also of practical relevance to the field of operations research; for instance, a strong influence of SFTA on performance would call for operations research tools to be used to increase the ability of stock-flow thinking (as proposed by Sterman (2010), or to reduce this cognitive bias (as suggested by Hämäläinen et al. (2013). On the contrary, a strong influence of GCA on performance would prompt operations researchers to develop tools that moderate its impact on performance. In addition, investigating the shape of the relation between GCA and performance is practically relevant, as an improved understanding of this relation would, for instance, allow us to improve tools supporting personnel scheduling – an area that is frequently explored by the operations research community (Van den Bergh, Beliën, De Bruecker, Demeulemeester, & De Boeck, 2013). For example, the tradeoff between performance losses via use of personnel with lower abilities, and cost savings because of lower wages, could be managed.

We aim to answer the research questions by setting up an experiment in which participants are placed in a typical dynamic production and inventory control setting similar to that used by Strohhecker and Größler (2013). The participants have to make repeated decisions on production quantities of a nonperishable good that is put in inventory, and from there delivered to business customers. Following related behavioral research (e.g., Moritz et al., 2013, Narayanan and Moritz, 2015), performance is measured as an outcome orientation using cumulated profit and various cost measures as alternate proxies.

The remainder of this article is organized as follows. First, we elaborate on the theoretical background from which our hypotheses are derived. Second, we describe our method, including our experimental design, measures, participants, and procedure. Third, we present and discuss our results. Finally, we describe our study's limitations and future research avenues.

Section snippets

Operational control, performance, and SFTA

Größler et al. (2008) emphasized that accumulation processes, delays, and feedback loops are widespread phenomena in operational systems that have not always been considered appropriately by managers who dynamically control these systems. From a structural system dynamics perspective, these three phenomena emerge from stocks and flows as building blocks, which are connected by information links (Forrester, 1961, Sterman, 2000). Executing dynamic first-order control in such a system, as shown in

Experimental design and measures

An observational research design with five observations obtained in a laboratory setting, plus additional data retrieved from archival sources, was deemed appropriate to test our research model including H1 to H3.2, as shown in Fig. 2 (Trochim & Donnelly, 2007).

The dependent variable, performance, was measured following the well-established experimental paradigm for investigating dynamic decision making (Brehmer, 1992) using a computer-simulated micro world. The micro world was nearly identical

Results

Table 1 shows the descriptive statistics and correlations between all variables that we investigated. We find highly significant medium to strong bivariate correlations for GCA, SFTA, and P that do not differ much between parametric and nonparametric methods.

To test our hypotheses, we calculated the research model as proposed in Fig. 2. The results are depicted in Fig. 8.

Based on the results shown in Fig. 8, we find support for H1, suggesting that SFTA indeed directly influences performance.

Theoretical contributions

From a theoretical angle, our analysis targets the influence of individual factors on operational decision-making performance in a dynamic stock-flow system that is presented as a typical production control environment. By finding significant support for our mediating model (including concavity), we contribute to behavioral operational research in multiple ways.

First, we provide evidence regarding the specific cognitive ability, SFTA, showing that it is an important predictor of performance in

References (104)

  • D. Frisch

    Reasons for framing effects

    Organizational Behavior & Human Decision Processes

    (1993)
  • Y. Ganzach et al.

    General mental ability and pay: Nonlinear effects

    Intelligence

    (2013)
  • R.P. Hämäläinen et al.

    On the importance of behavioral operational research: The case of understanding and communicating about dynamic systems

    European Journal of Operational Research

    (2013)
  • M. Holweg et al.

    Supply chain simulation – A tool for education, enhancement and endeavour

    International Journal of Production Economics

    (2002)
  • B. Hu et al.

    Strategy map concepts in a balanced scorecard cockpit improve performance

    European Journal of Operational Research

    (2017)
  • L. Larsen et al.

    The stability of general intelligence from early adulthood to middle-age

    Intelligence

    (2008)
  • N. Liberman et al.

    The effect of temporal distance on level of mental construal

    Journal of Experimental Social Psychology

    (2002)
  • K.S. McGrew

    CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research

    Intelligence

    (2009)
  • B.B. Moritz et al.

    Individual differences in the newsvendor problem: Behavior and cognitive reflection

    Journal of Operations Management

    (2013)
  • R.M. O'Keefe

    Experimental behavioural research in operational research: What we know and what we might come to know

    European Journal of Operational Research

    (2016)
  • L.M. Penney et al.

    A review of personality and performance: Identifying boundaries, contingencies, and future research directions

    Human Resource Management Review

    (2011)
  • J.D. Sterman

    Misperceptions of feedback in dynamic decision making

    Organizational Behavior and Human Decision Processes

    (1989)
  • J. Strohhecker et al.

    Do personal traits influence inventory management performance? The case of intelligence, personality, interest and knowledge

    International Journal of Production Economics

    (2013)
  • J. Van den Bergh et al.

    Personnel scheduling: A literature review

    European Journal of Operational Research

    (2013)
  • E.G. Anderson et al.

    The “physics” of capacity and backlog management in service and custom manufacturing supply chains

    System Dynamics Review

    (2005)
  • S. Axsäter

    Inventory control

    (2015)
  • J.F. Beckmann et al.

    Complex problem solving, intelligence, and learning ability

  • G.E. Bolton et al.

    Learning by doing in the newsvendor problem: A laboratory investigation of the role of experience and feedback

    Manufacturing & Service Operations Management

    (2008)
  • L. Booth Sweeney et al.

    Bathtub dynamics: Initial results of a systems thinking inventory

    System Dynamics Review

    (2000)
  • C.F. Camerer et al.

    The effects of financial incentives in experiments: A review and capital-labor-production framework

    Journal of Risk and Uncertainty

    (1999)
  • J.B. Carroll

    Human cognitive abilities: a survey of factor-analytic studies

    (1993)
  • R.B. Cattell

    Theory of fluid and crystallized intelligence: A critical experiment

    Journal of Educational Psychology

    (1963)
  • R.B. Cattell

    Intelligence: its structure, growth and action

    (1987)
  • J. Cohen et al.

    Applied multiple regression/correlation analysis for the behavioral sciences

    (2003)
  • P.T. Costa et al.

    Revised NEO personality inventory (NEO PI-R) and NEO five-factor inventory (NEO-FFI)

    (1992)
  • M. Cronin et al.

    Understanding the building blocks of system dynamics

    System Dynamics Review

    (2007)
  • R. Croson et al.

    Impact of POS data sharing on supply chain management: An experimental study

    Production and Operations Management

    (2003)
  • R. Croson et al.

    Upstream versus downstream information and its impact on the bullwhip effect

    System Dynamics Review

    (2005)
  • R. Croson et al.

    Behavioral causes of the bullwhip effect and the observed value of inventory information

    Management Science

    (2006)
  • R. Croson et al.

    Order stability in supply chains: Coordination risk and the role of coordination stock

    Production and Operations Management

    (2014)
  • D. Dörner

    The logic of failure. Strategic thinking for complex situations

    (1996)
  • J.W. Forrester

    Industrial dynamics: A major breakthrough for decision makers

    Harvard Business Review

    (1958)
  • J.W. Forrester

    Industrial dynamics

    (1961)
  • S. Frederick

    Cognitive reflection and decision making

    Journal of Economic Perspectives

    (2005)
  • M.C. Frey et al.

    Scholastic assessment or g? The relationship between the scholastic assessment test and general cognitive ability

    Psychological Science

    (2004)
  • A. Furnham

    Personality and intelligence at work: exploring and explaining individual differences at work

    (2008)
  • S. Geiser et al.

    UC and the SAT: Predictive validity and differential impact of the SAT I and SAT II at the University of California

    Educational Assessment

    (2002)
  • P. Gonçalves et al.

    The impact of endogenous demand on push-pull production systems

    System Dynamics Review

    (2005)
  • A. Größler

    Don't let history repeat itself – Methodological issues concerning the use of simulators in teaching and experimentation

    System Dynamics Review

    (2004)
  • A. Größler et al.

    System dynamics as a structural theory in operations management

    Production and Operations Management

    (2008)
  • Cited by (0)

    View full text