Elsevier

Information & Management

Volume 38, Issue 3, January 2001, Pages 153-165
Information & Management

A process model cognitive biasing effects in information systems development and usage

https://doi.org/10.1016/S0378-7206(00)00062-8Get rights and content

Abstract

There are many opportunities for failure when designing information systems. Most of the IS research investigating this area has concentrated on problems related to organizational environment, informational characteristics, or physical software design. Little attention has been given to the subconscious cognitive biases which can impact the manner in which information systems are developed and utilized. While it is recognized that elimination of these biases is neither feasible nor altogether desirable, it is further acknowledged that ignoring them can lead to an exacerbation of their undesirable effects. This paper provides a general framework to identify where system designer and decision maker biases are likely to occur, and suggests some moderating measures which might be taken to alleviate some of the potentially harmful effects of these biases.

Introduction

Under conditions of uncertainty, a decision maker (DM) is likely to search for available information from a vast array of sources and to accept information which conforms to some perception of a ‘reasonable’ outcome. Decision aids, such as decision support systems, are intended to reduce uncertainty and promote effective, objective solutions. Nonetheless, the retrieval and processing of information may be limited or distorted by the DM’s inherent cognitive biases. Further, given the involvement of a systems designers with their own set of cognitive biases, biased outcomes may actually be promoted since the information system was constructed based on the DM’s stated specifications.

Cognitive biases are neither readily apparent nor easily counterbalanced since they are developmentally inherent in the cognitive schemas and heuristics that form the basis of human information processing. These schemas and heuristics constitute behavioral patterns which dictate the way in which the DM approaches a problem. Such cognitive processes serve to reduce and combine mentally cumbersome quantities of information. They include filtering mechanisms applied in determining the elements or variables associated with the problem space, the analysis techniques used to examine the interrelationships between the components, and the manner in which outcomes are processed in relation to functional objectives.

While permitting the DM to manage complex information, numerous untoward consequences of cognitive heuristics and biases in decision making and information systems (IS) construction have been recognized by researchers for some time [4], [29], [34], [39], [42] (see [33] for a thorough review). More recently, the negative impacts of judgmental biases in day-to-day software development has been given attention by practitioners, especially with regard to testing and debugging [12], [37], [38].

Researchers generally agree that it would be futile to attempt to nullify biases, given their number, variety, and overlap with other biases, as well as the fact that they have can have positive utility in processing information. Furthermore, such efforts could possibly be counterproductive if they force the DM to alter successful patterns of behavior. However, in order to facilitate improved decision making, IS designers should understand the nature of such biases and the conditions under which they can be problematic. Counter-measures to alleviate some of the negative effects could subsequently be provided.

Some studies have offered suggestions for dealing with cognitive biases during IS design. These studies have focused on special problems of probabilistic information [8], [9], ways to incorporate control mechanisms to counter individual biases [4], [33], [37], or have focused on specific IS types [23], [29]. Although it has been suggested that a general model detailing the potential impacts of cognitive biases on IS design and usage is needed [4], [33], little effort has been directed toward this end.

The primary purpose of this paper is to consolidate and summarize the various biases that have been identified in various fields such as decision making, cognitive psychology and information systems. We are primarily concerned with biases which tend to arise in the development and use of small-scale IS. By that we mean those that are generally produced by a single designer, or under the direction of a single analyst, and are used by individual decision makers. This is in contrast to the larger, organization-wide systems which typically involve large teams of systems analysts to develop and are often used by teams of decision makers. While these latter systems can give rise to the same biases, because they are produced and used by groups of individuals, the situations which promote the biases, as well as how the biases manifest themselves, can vary.

It is not our intent to suggest that all biases are detrimental and should, therefore, be moderated. As shall be discussed, these biases underlie the cognitive schemas and heuristics which compose an IS designer’s and DM’s information processing patterns. Attempting to entirely alter the manner in which these individuals behave and reason is not only counterproductive, but also infeasible. However, dependent upon the available supply of information available, the information system derived from it, and the task environment, many of the biases can have detrimental affects. Our objective is to make the participants in this process, especially the IS designer, aware of the potential biases which exist, and what affect they could have on the decision making process. Consequently, the designer might be able to incorporate moderating strategies in developing the IS to alleviate some of unfavorable affects of biases (to be discussed later).

In Section 2, we review some conceptual foundations and past findings, drawing heavily from the psychology and IS literature. We next develop a process model to examine the manner in which the inherent behavioral schemas of the DM and the IS designer might filter or distort information during IS development, access, and application. A number of postulated biases are categorized in terms of their impact on the activities involved in the decision making process. In the Section 4, we discuss some of the theoretical constructs underlying cognitive processing mechanisms. Next, we propose a process model that illustrates some of the potential biases which may occur at various stages of the IS development and decision making process. We then suggest some preventive measures to alleviate some of the negative effects associated with these biases, and provide an example of how these measures might impact decision making. Finally, we offer some recommendations for future research.

Section snippets

Conceptual foundations and past findings

A number of theories have been advanced to describe human information processing. Most view cognitive processes as a set of information filtering and organizing mechanisms [6]. Because humans are inundated with enormous amounts of stimulus information, they tend to utilize a variety of mechanisms to cope with the environment. Coping mechanisms include chunking of information [34], [35], the use of heuristics [42], programmed strategies [18], and various other information categorization

A model of cognitive biases in information processing

The model, presented in Fig. 1, approaches the biases in information processing from two perspectives. It considers the two primary participants involved in development and usage of the information system, the IS designer and the DM (represented as shaded rectangles in the model), and three major components which impact the decision making process (represented as circles). These components include:

  • 1.

    The ‘Real-World’ universe of information, or Information supply, consisting of all facts and

Designer biases

Ideally, the IS designer should provide an objective analysis of the problem based on user requirements. In a large-scale, organization-wide IS, a team approach, relying on top–down systems analysis and design methodologies would likely be employed. As noted earlier, our focus is on smaller systems, where deployment of large systems analysis and design teams is not feasible.

For smaller systems, the IS designer can succumb to several biases. Many of these correspond to those noted for the DM

Decision maker biases

There are three linkages (labeled L3–L5 on Fig. 1) which may give way to a number of DM biases. Since these involve a variety of complex activities, the total number of potential biases postulated in considerably larger. The IS designer should be aware of the DM’s biases in order to construct an IS which moderate some of the negative impacts.

The linkages postulated for the DM are grouped according to the interface between the DM and the two major components of the information processing system

Moderation of potential biases

As mentioned previously, it is neither feasible nor desirable to attempt to eliminate all of the biases listed above. In some cases, however, it may desirable and possible to alleviate some of their potentially negative impacts.

There has been considerable research conducted, and prescriptions suggested, on techniques for avoiding biases, and di-biasing those that do occur [10], [16], [19], [21], [43]. Sage [33] has outlined the 10 most commonly mentioned strategies (see Table 2 for summary):

  • 1.

Conclusion

This paper is intended to provide an overview of some of the IS designer and DM biases which can occur at various phases in the IS development and information processing cycle. The processing model presented is intended as a mechanism for categorizing bias occurrences according to the activities which take place in the information processing cycle.

Our main intent is to make the participants in the systems development and usage process, especially the IS Designer, aware of the potentially

References (47)

  • W. Edwards et al.

    Probabilistic information processing systems: design and evaluation

    IEEE Transactions in Systems, Science and Cybernetics (SSC)

    (1960)
  • H.J. Einhorn, Learning from experience and suboptimal rules in decision processing, in: T.S. Wallsten (Ed.), Cognitive...
  • H.J. Einhorn, Overconfidence in judgment, in: R.A. Shweder (Ed.), Fallible Judgment in Behavioral Research, Jossey...
  • M. Eisenstadt, Tales of debugging from the front lines, in: C.R. Cook, J.C. Scholz, J.C. Spohrer (Eds.), Proceedings of...
  • B. Fischhoff

    Fault trees: sensitivity of estimated failure probabilities to problem presentation

    Journal of Experimental Psychology: Human Perception and Performance

    (1978)
  • B. Fischoff, For those condemned to study the past: reflections on historical judgment, in: R.A. Shweder (Ed.),...
  • D.L. Hamilton, Cognitive biases in the perception of social groups, in: J.S. Carroll, J.W. Payne (Eds.), Cognition and...
  • R.M. Hogarth et al.

    Forecasting and planning: an evaluation

    Management Science

    (1981)
  • R.M. Hogarth, Judgment and Choice: The Psychology of Decisions, Wiley, New York,...
  • G.P. Huber, The nature of organizational decision making and the design of decision support systems, MIS Quarterly...
  • E.E. Jones, G.R. Goethals, Order effects in impression formation: attribution context and the nature of the entity, in:...
  • L.R. Keller

    The effects of problem representation on the sure thing and substitution principles

    Management Science

    (1985)
  • D. Kahneman et al.

    On the psychology of prediction

    Psychology Review

    (1973)
  • Cited by (26)

    • People with jumping to conclusions bias tend to make context-independent decisions rather than context-dependent decisions

      2022, Consciousness and Cognition
      Citation Excerpt :

      Cognitive biases have been observed in the general population. Due to information processing limitations, people need to use a variety of information simplification mechanisms that would help them process and interpret enormous amounts of data they encounter every day and take decisions under a wide range of complex, uncertain situations (Kirs et al., 2001). Although there are often biases in human judgment and decision-making, not all people exhibit these frequently dysfunctional information processing patterns.

    • Relations between Cognitive Biases and Some Concepts of Information Behavior

      2020, Data and Information Management
      Citation Excerpt :

      In information science, while cognitive biases have not received much attention in the domain of human information behavior, there have been studies on relevant concepts such as uncertainty (e.g., works by Kuhlthau, 1993; Ingwersen, 1996; and Wilson, Ford, Ellis, Foster, & Spink, 2002), which plays an important role in information behavior. Cognitive biases have received more attention from researchers in many areas, such as information systems (e.g., Kirs, Pflughoeft, & Kroeck, 2001; Arnott, 2006) and health information (e.g., Lau & Coiera, 2007, 2009; Schweiger, Oeberst, & Cress, 2014). There are two reviews of research on cognitive biases in information systems (Fleischmann, Amirpur, Benlian, & Hess, 2014; Mohanani, Salman, Turhan, Rodriguez, & Ralph, 2017).

    • Misrepresentations and Criminal Liability in Project Reporting: A Case Study of the Failed Virgil C. Summer Project

      2023, Journal of Legal Affairs and Dispute Resolution in Engineering and Construction
    • Complied by Belief Consistency: The Cognitive-Information Lens of User-Generated Persuasion

      2023, Journal of Theoretical and Applied Electronic Commerce Research
    View all citing articles on Scopus
    View full text