Skip to main content
SpringerLink
Log in

The CACHE Study: Group Effects in Computer-supported Collaborative Analysis

  • Published:
Computer Supported Cooperative Work (CSCW) Aims and scope Submit manuscript

Abstract

The present experiment investigates effects of group composition in computer-supported collaborative intelligence analysis. Human cognition, though highly adaptive, is also quite limited, leading to systematic errors and limitations in performance – that is, biases. We experimentally investigated the impact of group composition on an individual’s bias, by composing groups that differ in whether their members initial beliefs are diverse (heterogeneous group) or similar (homogeneous group). We study three-member, distributed, computer-supported teams in heterogeneous, homogeneous, and solo (or nominal) groups. We measured bias in final judgment, and also in the selection and evaluation of the evidence that contributed to the final beliefs. The distributed teams collaborated via CACHE-A, a web-based software environment that supports a collaborative version of Analysis of Competing Hypotheses (or ACH, a method used by intelligence analysts). Individuals in Heterogeneous Groups showed no net process cost, relative to noninteracting individuals. Both heterogeneous and solo (noninteracting) groups debiased strongly, given a stream of balanced evidence. In contrast, individuals in Homogenous Groups did worst, accentuating their initial bias rather than debiasing. We offer suggestions about how CACHE-A supports collaborative analysis, and how experimental investigation in this research area can contribute to design of CSCW systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6

Similar content being viewed by others

References

  • Adelman, L., Tolcott, M.A. and T.A. Bresnick (1993): Examining the effect of information order on expert judgment, Organizational Behavior and Human Decision Processes, vol. 56, pp. 348–369.

    Article  Google Scholar 

  • Arnott, D. (2006): Cognitive Biases and Decision Support Systems Development: A Design Science Approach. Information Systems Journal, vol. 16(1), pp. 55–78 doi:10.1111/j.1365-2575.2006.00208.x.

    Article  Google Scholar 

  • Benbasat, I. and J. Lim (2000): Information Technology Support For Debiasing Group Judgments: An Empirical Evaluation. Organizational Behavior and Human Decision Processes, vol. 83, pp. 167–183 doi:10.1006/obhd.2000.2905.

    Article  Google Scholar 

  • Benjamin, K. (1990): Why We Still Use Our Heads Instead of Formulas: Toward an Integrative Approach. Psychological Bulletin, vol. 107(3), pp. 296–310 doi:10.1037/0033-2909.107.3.296.

    Article  Google Scholar 

  • Bornstein, B.H. and A.C. Emler (2001): Rationality in Medical Decision Making: A Review of the Literature On Doctors’ Decision-making Biases. Journal of Evaluation in Clinical Practice, vol. 7(2), pp. 97–107 doi:10.1046/j.1365-2753.2001.00284.x.

    Article  Google Scholar 

  • Camerer, C.F. and E.J. Johnson (1991): The Process–Performance Paradox in Expert Judgment: How Can Experts Know so Much and Predict so Badly? In K.A. Ericsson and J. Smith (eds): Towards a General Theory of Expertise: Prospects and LimitsNew York: Cambridge University Press, pp. 195–217.

    Google Scholar 

  • Card, S.K. (2005): The Science of Analytical Reasoning. In J.J. Thomas and K.A. Cook (eds): Illuminating the Path: the Research and Development Agenda for Visual AnalyticsIEEE CS: Los Alamitos, CA.

    Google Scholar 

  • Chapman, G.B., G.R. Bergus and A.S. Elstein (1996): Order of Information Affects Clinical Judgment. Journal of Behavioral Decision Making, vol. 9(3), pp. 201–211 doi:10.1002/(SICI)1099-0771(199609)9:3<201::AID-BDM229>3.0.CO;2-J.

    Article  Google Scholar 

  • Cheikes, B.A., M.J. Brown, P.E. Lehner and L. Alderman (2004): Confirmation Bias in Complex Analyses. Technical Report No. MTR 04B0000017. Bedford, MA: MITRE.

    Google Scholar 

  • Cho, H.-K. (2004): The effect of Delphi Structure on Small and Medium-sized Asynchronous Groups. Ph. D. Dissertation Thesis, New Jersey Institute of Technology, Information Systems Department.

  • Cho, H.-K. and M. Turoff (2001): Debiasing Group Judgments through Computerized Delphi Systems. In Proceedings of AMCIS 2001, August, Boston, MA.

  • Convertino, G., H.M. Mentis, P. Bhambare, C. Ferro, J.M. Carroll and M.B. Rosson (2008): Comparing Media in Emergency Planning. In Proceedings of the 5th International ISCRAM. Conference. Washington, DC, USA, May 4–7, 2008.

  • Cook, M.B. and H.S. Smallman (2007): Collaborative intelligence analysis: Debiasing through graphical evidence layout. In: Proceedings of the 51st Annual Meeting of the Human Factors and Ergonomics Society, Baltimore, MD, pp. 16–20.

  • Cummings, J.N. (2004): Work Groups, Structural Diversity, and Knowledge Sharing in a Global Organization. Management Science, vol. 50(3), pp. 352–364.

    Article  MathSciNet  Google Scholar 

  • Davis, E.B. and R.H. Ashton (2002): Threshold Adjustment in Response to Asymmetric Loss Functions: The Case of Auditors’ “Substantial Doubt” Thresholds. Organizational Behavior and Human Decision Processes, vol. 89, pp. 1082–1099 doi:10.1016/S0749-5978(02)00009-2.

    Article  Google Scholar 

  • Dennis, A.R. and J.S. Valacich (1993): Computer Brainstorms: More Heads are Better Than One. The Journal of Applied Psychology, vol. 78(4), pp. 531–537 doi:10.1037/0021-9010.78.4.531.

    Article  Google Scholar 

  • Dennis, A.R., K.M. Hilmer and N.J. Taylor (1997): Information Exchange and Use in GSS and Verbal Group Decision Making: Effects of Minority Influence. Journal of Management Information Systems Archive, vol. 14(3), pp. 61–88.

    Article  Google Scholar 

  • Fjermestad, J. and S.R. Hiltz (2001): An Assessment of Group Support Systems Research: Methodology. Journal of Management Information Systems, vol. 15(3), pp. 7–149.

    Google Scholar 

  • Fugelsang, J.A., C.B. Stein, A.E. Green and K.N. Dunbar (2004): Theory and Data Interactions of the Scientific Mind: Evidence From the Molecular and the Cognitive Laboratory. Canadian Journal of Experimental Psychology, vol. 58(2), pp. 86–95 doi:10.1037/h0085799.

    Google Scholar 

  • Gallupe, B.R., L.M. Bastianutti and W.H. Cooper (1991): Unblocking Brainstorms. The Journal of Applied Psychology, vol. 76, pp. 137–142 doi:10.1037/0021-9010.76.1.137.

    Article  Google Scholar 

  • George, J.F., K. Duffy and M. Ahuja (2000): Countering the Anchoring and Adjustment Bias with Decision Support Systems. Decision Support Systems, vol. 29(2), pp. 195–206 doi:10.1016/S0167-9236(00)00074-9.

    Article  Google Scholar 

  • Gettys, C.F., C. Kelly III and C.R. Peterson (1982): The Best-guess Hypothesis in Multistage Inference. In D. Kahneman, P. Slovic and A. Tversky (eds): Judgment Under Undertainty: Heuristics and BiasesNew York: Cambridge University Press, pp. 370–377.

    Google Scholar 

  • Gigone, D. and R. Hastie (1996): The Impact of Information on Small Group Choice. Journal of Personality and Social Psychology, vol. 72, pp. 132–140 doi:10.1037/0022-3514.72.1.132.

    Article  Google Scholar 

  • Heuer, Richards J. Jr. (1999): The Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency.

    Google Scholar 

  • Hightower, R. and L. Sayeed (1995): The Impact of Computer-mediated Communication Systems on Biased Group Discussion. Computers in Human Behavior, vol. 11(1), pp. 33–44 doi:10.1016/0747-5632(94)00019-E.

    Article  Google Scholar 

  • Hogarth, R.M. and Hillel J. Einhorn (1992): Order effects in belief updating: The belief-adjustment model. Cognitive Psychology, vol. 24(1), pp. 1–55 doi:10.1016/0010-0285(92)90002-J.

    Article  Google Scholar 

  • Johnston, R. (2003): Reducing Analytic Error: Integrating Methodologists into Teams of Substantive Experts. Studies in Intelligence, vol. 47(1), pp. 57–65.

    Google Scholar 

  • Johnston, R. (2005): Analytic Culture in the U.S. Intelligence Community: An Ethnographic Study. Washingon, DC: Central Intelligence Agency, Center for the Study of Intelligence.

    Google Scholar 

  • Johnson, E.M. and S.M. Halpin (1974): Multistage Inference Models For Intelligence Analysis. Report AD 785-639. Arlington, VA: Army Research Institute for the Behavioral and Social Sciences(Distributed by US Department of Commerce, Springfield, VA).

    Google Scholar 

  • Kahneman, D., P. Slovic and A. Tversky (eds) (1982): Judgment under Uncertainty: Heuristics and Biases. New York: Cambridge University Press.

    Google Scholar 

  • Keefer, Donald L., Craig W. Kirkwood and James L. Corner (2004): Perspective on Decision Analysis Applications, 1990–2001. Decision Analysis, vol. 1(1), pp. 4–22 doi:10.1287/deca.1030.0004.

    Article  Google Scholar 

  • Kerr, N.L. and S.R. Tindale (2004): R.S. Group Performance and Decision Making. Annual Review of Psychology, vol. 55, pp. 623–655 doi:10.1146/annurev.psych.55.090902.142009.

    Article  Google Scholar 

  • Kerr, Norbert L., R.J. MacCoun and G.P. Kramer (1996): Bias in Judgment: Comparing Individuals and Groups. Psychological Review, vol. 103(4), pp. 687–719 doi:10.1037/0033-295X.103.4.687.

    Article  Google Scholar 

  • Kerstholt, J.H. and J.L. Jackson (1998): Judicial Decision Making: Order of Evidence Presentation and Availability of Background Information. Applied Cognitive Psychology, vol. 12, pp. 445–454.

    Article  Google Scholar 

  • Kiesler, S. and L. Sproull (1992): Group Decision Making and Communication Technology. Organizational Behavior and Human Decision Processes, vol. 52, pp. 96–123 doi:10.1016/0749-5978(92)90047-B.

    Article  Google Scholar 

  • Klayman, J. and Y.-W. Ha (1987): Confirmation, Disconfirmation, and Information in Hypothesis Testing. Psychological Review, vol. 94(2), pp. 211–228 doi:10.1037/0033-295X.94.2.211.

    Article  Google Scholar 

  • Kraut, R.E. (2003): Applying Social Psychological Theory to the Problems of Group Work. In John M. Carroll (ed): HCI Models, Theories, and Frameworks: Toward a Multidisciplinary ScienceNew York: Morgan Kaufmann, pp. 325–356.

    Chapter  Google Scholar 

  • Larson, J.R. Jr., P.G. Foster-Fishman and C.B. Keys (1994): Discussion of Shared and Unshared Information in Decision-making Groups. Journal of Personality and Social Psychology, vol. 67, pp. 446–461 doi:10.1037/0022-3514.67.3.446.

    Article  Google Scholar 

  • Lim, L.-H. and I. Benbasat (1997): The Debiasing Role of Group Support Systems: An Experimental Investigation of the Representativeness Bias. Int. J. Human–Computer Studies, vol. 47, pp. 453–471 doi:10.1006/ijhc.1997.0137.

    Article  Google Scholar 

  • Lipshitz, R., G. Klein, J. Orasanu and E. Salas (2001): Taking stock of naturalistic decision making. Journal of Behavioral Decision Making, vol. 14(5), pp. 331–352 doi:10.1002/bdm.381.

    Article  Google Scholar 

  • Neale, D.C., J.M. Carroll and R.M. Beth (2004): Evaluating Computer-supported Cooperative WOrk: Models And Frameworks. In Proceedings of the ACM conference on Computer Supported Cooperative Work, Chicago, IL.

  • Nunamaker, J.F. Jr., A.R. Dennis, J.S. Valacich, D.R. Vogel and J.F. George (1991): Electronic Meeting Systems to Support Group Work. Communications of the ACM, vol. 34(7), pp. 40–61.

    Article  Google Scholar 

  • Perrin, B.M., B.J. Barnett, L. Walrath and J.D. Grossman (2001): Information Order and Outcome Framing: An Assessment of Judgment Bias, In a Naturalistic Decision-Making Context. Human Factors, vol. 43(2), pp. 227–238 doi:10.1518/001872001775900968.

    Article  Google Scholar 

  • Pirolli, P., T. Lee and Stuart K. Card (2004): Leverage Points for Analyst Technology Identified through Cognitive Task Analysis Technical Report. Palo Alto, CA: PARC.

    Google Scholar 

  • Pirolli, P., L. Good, J. Heiser, J. Shrager and S. Hutchins (2005): UIR Technical Report. Palo Alto, CA: PARC.

    Google Scholar 

  • Reagan-Cirincione, P. (1994): Improving the accuracy of group judgment: a process intervention combining group facilitation, social judgment analysis, and information technology. Organizational Behavior and Human Decision Processes, vol. 58, pp. 246–270 doi:10.1006/obhd.1994.1036.

    Article  Google Scholar 

  • Russell, D.M., M.J. Stefik, P. Pirolli, and S.K. Card (1993): The Cost Structure of Sensemaking. In: Proceedings of the SIGCHI conference on Human factors in computing systems, April 24–29, 1993, Amsterdam, The Netherlands, pp. 269–276.

  • Scholtz, J., E. Morse and P.S. Michelle (2006): Evaluation metrics and methodologies for user-centered evaluation of intelligent systems. Interacting with Computers, vol. 18, pp. 1186–1214 doi:10.1016/j.intcom.2006.08.014.

    Article  Google Scholar 

  • Schultz-Hart, S., D. Frey, C. Lüthgens and S. Moscovici (2000): Biased Information Search in Group Decision Making. Journal of Personality and Social Psychology, vol. 78(4), pp. 655–669 doi:10.1037/0022-3514.78.4.655.

    Article  Google Scholar 

  • Shanteau, J. (1992): Competence in Experts: The Role of Task Characteristics. Organizational Behavior and Human Decision Processes, vol. 53, pp. 252–266 doi:10.1016/0749-5978(92)90064-E.

    Article  Google Scholar 

  • Shrager, J. (2005): CACHE; The Collaborative Analysis of Competing Hypotheses Environment [computer software]. Palo Alto, CA: Xerox Palo Alto Research Center.

    Google Scholar 

  • Shrager, J., D. Billman, G. Convertino, J.P. Massar and P. Pirolli (in review): CACHE: Web-based support for distributed multi-stage inference.

  • Smallman, H.S. (2008): JIGSAW – Joint Intelligence Graphical Situation Awareness Web for Collaborative Intelligence Analysis. In M.P. Letsky, N.W. Warner, S.M. Fiore and C.A.P. Smith (eds.): Macrocognition in Teams: Theories and Methodologies. Ashgate Publishing Limited, Hampshire, England, pp. 321–337 (in press).

  • Stasser, G. and W. Titus (2003): Hidden Profiles: A Brief History. Psychological Inquiry, vol. 14(3–4), pp. 302–311.

    Google Scholar 

  • Straus, S.G. and J.E. McGrath (1994): Does the Medium Matter? The Interaction of Task Type and Technology on Group Performance and Member Reactions. The Journal of Applied Psychology, vol. 79(1), pp. 87–89 doi:10.1037/0021-9010.79.1.87.

    Article  Google Scholar 

  • Tolcott, M.A., F.F. Marvin and P.E. Lehner (1989): Expert Decisionmaking in Evolving Situations. IEEE Transact. on Systems. Man, and Cybernetics, vol. 19(3), pp. 606–615 doi:10.1109/21.31066.

    Article  Google Scholar 

  • Trent, S.A., E.S. Patterson and D.D. Woods (2007): Challenges for Cognition. J. of Cognitive Engineering and Decision Making, vol. 1(1), pp. 75–97.

    Google Scholar 

  • Tversky, A. and D. Kahneman (1974): Judgments under Uncertainty. Heuristics and Biases. Science, vol. 185, pp. 1124–1131 doi:10.1126/science.185.4157.1124.

    Article  Google Scholar 

  • van Knippenberg, D. and M.C. Schippers (2007): Work Group Diversity. Annual Review of Psychology, vol. 58, pp. 515–541 doi:10.1146/annurev.psych.58.110405.085546.

    Article  Google Scholar 

  • Wason, P.C. (1960): On the Failure to Eliminate Hypotheses in a Conceptual Task. The Quarterly Journal of Experimental Psychology, vol. 12, pp. 129–140 doi:10.1080/17470216008416717.

    Article  Google Scholar 

Download references

Acknowledgements

This research was funded by the Novel Intelligence from Massive Data program, under contract no. MDA904-03-C-0404 and by the Office of Naval Research no. N00014-96-C-0097. We thank Stuart Card and the UIR researchers at PARC for their feedback, and MITRE Corporation, Naval Postgraduate School, PARC employees, and Stanford students for contributing to this study.

Author information

Authors and Affiliations

  1. College of Information Sciences and Technology, Penn State University, University Park, PA, 16802, USA

    Gregorio Convertino

  2. Palo Alto Research Center, 3333 Coyote Hill Road, Palo Alto, CA, 94304, USA

    Dorrit Billman & Peter Pirolli

  3. Berkeley, CA, USA

    J. P. Massar

  4. Symbolic Systems Program, Stanford University, Stanford, CA, 94305, USA

    Jeff Shrager

  5. Center for Study of Language and Information, Stanford University, Cordura Hall, Rm 110, Stanford, CA, 94305, USA

    Dorrit Billman

Authors
  1. Gregorio Convertino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dorrit Billman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peter Pirolli
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. J. P. Massar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jeff Shrager
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dorrit Billman.

Appendix

Appendix

Additional screen shots show details of the CACHE-A interface. Examples show content from training phase.

Figure 7
figure 7

ACH Matrix row: tabular interface to specify weight or importance of one evidence item and its consistency relation to the three hypotheses.

Full size image
Figure 8
figure 8

Search tool. The two top figures show the search tool. In the tool, the analyst specifies a if to search over interpretations (top left) or items content (top right), b the search keywords, c the listing method: i.e., concise vs. full results, and d the keyword-result marching method (bottom left). A specific item can be viewed by clicking on the “[I]” (top left) link or “[E]” link (top right) at the bottom of the search tool. An evidence viewer (bottom right) will open for this item as a result.

Full size image
Figure 9
figure 9

Evidence viewer. The window displays the content of an evidence item and its interpretations. Each item has one default interpretation. The analyst can edit the current interpretation or add a new interpretation using the text field under the evidence text.

Full size image
Figure 10
figure 10

Ticker and partner’s matrix. The analyst can open read-only views of collaborators’ ACH matrices by clicking on the top links in the Ticker, while CACHE-A is in collaborative mode. The links “ALPHA1” and “BETA1” in the ticker, the top figure, give GAMMA1 access to her/his partner matrices. The bottom figure shows the read-only view of BETA1’s matrix that would open on her/his screen.

Full size image

Rights and permissions

Reprints and permissions

About this article

Cite this article

Convertino, G., Billman, D., Pirolli, P. et al. The CACHE Study: Group Effects in Computer-supported Collaborative Analysis. Comput Supported Coop Work 17, 353–393 (2008). https://doi.org/10.1007/s10606-008-9080-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10606-008-9080-9

Key words

Search

Navigation