Article

Evaluating computer-supported cooperative work: models and frameworks

Authors:

Dennis C. Neale,

John M Carroll,

Mary Beth RossonAuthors Info & Claims

CSCW '04: Proceedings of the 2004 ACM conference on Computer supported cooperative work

Pages 112 - 121

https://doi.org/10.1145/1031607.1031626

Published: 06 November 2004 Publication History

Abstract

Evaluating distributed CSCW applications is a difficult endeavor. Frameworks and methodologies for structuring this type of evaluation have become a central concern for CSCW researchers. In this paper we describe the problems involved in evaluating remote collaborations, and we review some of the more prominent conceptual frameworks of group interaction that have driven CSCW evaluation in the past. A multifaceted evaluation framework is presented that approaches the problem from the relationships underlying joint awareness, communication, collaboration, coordination, and work coupling. Finally, recommendations for carrying out multifaceted evaluations of remote interaction are provided.

References

[1]

Arrow, H., McGrath, J. E., and Berdahl, J. L. (2000). Small Groups as Complex Systems: Formation, Coordination, Development, and Adaptation. Thousand Oaks, CA: Sage.

[2]

Borghoff, U. M., and Schlichter, J. H. (2000). Computer-Supported Cooperative Work: Introduction to Distributed Applications: Springer.

Digital Library

[3]

Brooks, F. P. (1995). The Mythical Man-Mouth. Reading, Massachusetts: Addison-Wesley.

Digital Library

[4]

Carroll, J. M., Neale, D. C., Isenhour, P. L., Rosson, M. B., and McCrickard, D. S. (2003). Notification and awareness: synchronizing task-oriented collaborative activity. International Journal of Human-Computer Studies, 58, 605--632.

Digital Library

[5]

Carroll, J. M., Chin, G., Rosson, M. B., & Neale, D. C. (2000). The development of cooperation: Five years of participatory design in the virtual school. In Proceedings on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (pp. 239--251), New York: Association for Computing Machinery.

Digital Library

[6]

Clark, H. H. (1996). Using Language. New York: Cambridge University Press.

[7]

Convertino, G., Neale, D. C., Hobby, L., Carroll, J. M., and Rosson, M. B. (2004). A laboratory method for studying activity awareness. In NordiCHI 2004 (to appear). Tampere, Finland: Association of Computing Machinery.

Digital Library

[8]

Damianos, L., Hirschman, L., Kozierok, R., Kurtz, J., Greenberg, A., Walls, K., Laskowski, S., and Scholtz, J. (1999). Evaluation for collaborative systems. ACM Computing Surveys, 31(2).

Digital Library

[9]

Dourish, P., and Bellotti, V. (1992). Awareness and coordination in shared workspaces. In Proceedings of the ACM CSCW '92 Conference on Computer Supported Cooperative Work (pp. 107--113). New York: Association for Computing Machinery.

Digital Library

[10]

Ganoe, C. H., Somervell, J. P., Neale, D. C., Isenhour, P. L., Carroll, J. M., Rosson, M. B., and McCrickard, D. S. (2003). Classroom BRIDGE: Using collaborative public and desktop timelines to support activity awareness. In Proceedings of the ACM '03 Symposium on User Interface Software and Technology (UIST) (pp. 21--30). New York: Association for Computing Machinery.

Digital Library

[11]

Greene, J. C., Caracelli, V. J., and Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255--274.

[12]

Grudin, J. (1988). Why groupware applications fail: Problems in design and evaluation. Office: Technology and People, 4(3), 245--264.

[13]

Grudin, J. (2002). Group dynamics and ubiquitous computing. Communications of the ACM, 45(12), 74--78.

Digital Library

[14]

Hare, A. P. (1992). Groups, teams, and social interaction: Theories and applications. New York: Praeger.

[15]

Hartswood, M., and Procter, R. (2000). Design guidelines for dealing with breakdowns and repairs in collaborative work settings. International Journal of Human-Computer Studies, 53, 91--120.

Digital Library

[16]

Helms, J., Neale, D. C., and Carroll, J. M. (2000). Data logging: Higher-level capturing and multi-level abstracting of user activities. In Proceedings of the 40th Annual Meeting of the Human Factors and Ergonomics Society (pp. 303--306). Santa Monica, CA: Human Factors and Ergonomics Society.

[17]

Hendrick, H. W., and Kleiner, B. M. (2000). Macroergonomics: An introduction to work system design. (Vol. 2). Santa Monica, CA: Human Factors and Ergonomics Society.

[18]

Humphries, W., Neale, D. C., McCrickard, D. S., and Carroll, J. M. (2004). Laboratory Simulation Methods for Studying Complex Collaborative Tasks. In Proceedings of the 48th Annual Meeting of the Human Factors and Ergonomics Society. Human Factors and Ergonomics Society (to appear). Santa Monica, CA.

[19]

Isenhour, P. L., Carroll, J. M., Neale, D. C., Rosson, M. B., and Dunlap, D. R. (2000). The virtual school: An integrated collaborative environment for the classroom. Educational Technology and Society, Special Issue on "On-Line Collaborative Learning Environments", 3(3), http://ifets.ieee.org/periodical/.

[20]

Klein, G. (2001). Features of team coordination. In M. McNeese, E. Salas, and M. Endsley (Eds.), New Trends in Cooperative Activities: Understanding System Dynamics in Complex Environments (pp. 68--95). Santa Monica, CA: Human Factors and Ergonomics Society.

[21]

Malone, T. W., and Crowston, K. (1994). The interdisciplinary study of coordination. ACM Computing Surveys, 26(1), 87--119.

Digital Library

[22]

McGrath, J. E. (1984). Groups: Interaction and performance. Englewoods Cliffs, NJ: Prentice-Hall.

[23]

McGrath, J. E. (1990). Time matters in groups. In J. Galegher, R. E. Kraut, and C. Egido (Eds.), Intellectural Teamwork: Social and Technological Foundations of Cooperative Work (pp. 23--61). Hillsdale, N. J.: Lawrence Erlbaum.

Digital Library

[24]

McGrath, J. E. (1994). Methodology matters: Doing research in the behavioral and social sciences. In R. M. Baecker, J. Grudin, W. A. S. Buxton, and S. Greenberg (Eds.), Readings in human-computer interaction: Toward the year 2000 (pp. 152--169). San Francisco, CA: Morgan Kaufmann.

Digital Library

[25]

Miles, M. B., and Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage.

[26]

Nardi, B. A. E. (Ed.). (1996). Context and consciousness: Activity theory and human-computer interaction. Cambridge, MA: MIT Press.

Digital Library

[27]

Neale, D. C., and Carroll, J. M. (1999). Multi-faceted evaluation for complex, distributed activities. In Proceedings of CSCL'99 Computer Supported Cooperative Learning (pp. 425--433). Mahwah, New Jersy: Lawrence Erlbaum.

Digital Library

[28]

Neale, D. C., Dunlap, D. R., Isenhour, P., and Carroll, J. M. (2000). Collaborative critical incident development. In Proceedings of the 44th Annual Meeting of the Human Factors and Ergonomics Society (pp. 598--601). Santa Monica, CA: Human Factors and Ergonomics Society.

[29]

Olson, G. M., and Olson, J. S. (1997). Research on computer supported cooperative work. In M. Helander, T. K. Landauer, and P. Prablu (Eds.), Handbook of Human-Computer Interaction (pp. 1433--1456). Amsterdam, The Netherlands: Elsevier Science.

[30]

Olson, G. M., and Olson, J. S. (2000). Distance matters. Human-Computer Interaction, 15, 139--178.

Digital Library

[31]

Olson, G. M., and Olson, J. S. (2001). Technology support for collaborative workgroups. In G. M. Olson, T. W. Malone, and J. B. Smith (Eds.), Coordination Theory and Collaboration Technology (pp. 559--584). Mahwah, N. J.: Lawrence Erlbaum.

[32]

Pedhazur, E. J. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, New Jersey: Lawrence Erlbaum.

[33]

Pinelle, D., and Gutwin, C. (2000). A review of groupware evaluations. In Proceedings of the 9th IEEE WETICE (pp. 86--91).

Digital Library

[34]

Pinelle, D., Gutwin, C., and Greenberg, A. (2003). Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration. Transactions on Computer-Human Interaction, 10(4), 281--311.

Digital Library

[35]

Pinsonneault, A., and Kraemer, K. L. (1989). The impact of technological support on groups: An assessment of the empirical research. Decision Support Systems, 5(2), 197--211.

[36]

Plowman, L., Rogers, Y., and Ramage, M. (1995). What are workplace studies for?. In Proceedings of ECSCW '95 European Conference on Computer-Supported Cooperative Work (pp. 309--324). Dordrecht: Kluwer.

Digital Library

[37]

Ross, S., Ramage, M., and Rogers, Y. (1995). PETRA: Participatory evaluation through redesign and analysis. Interacting with Computers, 7(4), 335--360.

[38]

Ruhleder, K., and Jordan, B. (1998). Video-based interaction analysis (VBIA) in distributed settings: A tool for analyzing multiple-site, technology-supported interactions. In Proceeding of the Participatory Design Conference (pp. 195--196). Seattle, WA.

[39]

Salvador, T., Scholtz, J., and Larson, J. (1996). The denver model for groupware design. SIGCHI Bulletin, 28(1), 52--58.

Digital Library

[40]

Sproull, L., and Kiesler, S. (1991). Connections: New ways of working in the networked organization. London: MIT Press.

Digital Library

[41]

Stanney, K. M., and Maxey, J. (1997). Socially centered design. In G. Salvendy (Ed.), Handbook of Human Factors and Ergonomics (2nd Edition) (pp. 637--656). New York: Wiley.

[42]

Suchman, L. (1987). Plans and situated actions. New York: Cambridge University Press.

Digital Library

[43]

Tashakkori, A., and Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: SAGE.

[44]

Thomas, P. J. (Ed.). (1996). CSCW Requirements and Evaluation. London: Springer.

Digital Library

[45]

Twidale, M., Randall, D., and Bentley, R. (1994). Situated evaluation for cooperative systems. In Proceedings of the ACM CSCW '94 Conference on Computer Supported Cooperative Work (pp. 441--452). New York: Association for Computing Machinery.

Digital Library

[46]

Vicente, K. J. (1997). Heeding the legacy of Meister, Brunswik, & Gibson: Toward a broader view of human factors research. Human Factors, 39(2), 323--328.

[47]

Watts, L., and Monk, A. (1996). Inter-personal awareness and synchronization: Assessing the value of communication technologies. International Journal of Human-Computer Studies, 44, 849--873.

Digital Library

[48]

Whittaker, S., and Schwarz, H. (1999). Meetings of the board: The impact of scheduling medium on long term group coordination in software development. Computer Supported Cooperative Work, 8, 175--205.

Digital Library

[49]

Woods, D. D. (2003). Discovering how distributed cognitive systems work. In E. Hollnagel (Ed.), Handbook of cognitive task design (pp. 37--53). Mahwah, N.J.: Lawarence Erlbaum.

[50]

Xiao, Y., and Vicente, K. J. (2000). A framework for epistemological analysis in empirical (laboratory and field) studies. Human Factors, 42(1), 87--101.

Cited By

Omran LAndersson Hagiwara MPuaca GMaurin Söderholm H(2024)The impact of video consultation on interprofessional collaboration and professional roles: a simulation-based study in prehospital stroke chain of careJournal of Interprofessional Care10.1080/13561820.2024.234407538:4(664-674)Online publication date: 8-May-2024
https://doi.org/10.1080/13561820.2024.2344075
Nishide SNishida T(2024)Using Word Games as Facilitator to Awareness Raising Communication in Public SpacesCollaboration Technologies and Social Computing10.1007/978-3-031-67998-8_15(212-219)Online publication date: 20-Aug-2024
https://doi.org/10.1007/978-3-031-67998-8_15
Correia AGrover ASchneider DPimentel AChaves Rde Almeida MFonseca B(2023)Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine InteractionApplied Sciences10.3390/app1304219813:4(2198)Online publication date: 8-Feb-2023
https://doi.org/10.3390/app13042198
Show More Cited By

Index Terms

Evaluating computer-supported cooperative work: models and frameworks
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Computer supported cooperative work
2. Social and professional topics
  1. Professional topics
    1. Computing and business
      1. Computer supported cooperative work

Recommendations

Emotion understanding and performance during computer-supported collaboration

Individuals collaborating around and through computers benefit from receiving information that helps them understand one another, which is often termed awareness. This article explores what collaborators understand about each other's emotions and the ...
Differentiated Awareness-Support in Computer Supported Collaborative Work

Awareness has, for decades, been a topic of interest in CSCW research, as it is a critical condition for effective coordination of collaborative work. In this special issue we present papers that discuss overviews of awareness aspects as well as ...
Information exchanges patterns in a computer-supported cooperative work environment

Software for supporting cooperative work, or groupware, is an approach for increasing project team productivity by changing the way team members work and communicate with each other. Since groupware use can be influenced by various individual, team, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '04: Proceedings of the 2004 ACM conference on Computer supported cooperative work

November 2004

644 pages

ISBN:1581138105

DOI:10.1145/1031607

General Chairs:
Jim Herbsleb
Carnegie Mellon University, USA
,
Gary Olson
University of Michigan, USA

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 November 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CSCW04

Sponsor:

CSCW04: Computer Supported Cooperative Work

November 6 - 10, 2004

Illinois, Chicago, USA

Acceptance Rates

CSCW '04 Paper Acceptance Rate 53 of 176 submissions, 30%;

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

190
Total Citations
View Citations
6,298
Total Downloads

Downloads (Last 12 months)104
Downloads (Last 6 weeks)4

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Omran LAndersson Hagiwara MPuaca GMaurin Söderholm H(2024)The impact of video consultation on interprofessional collaboration and professional roles: a simulation-based study in prehospital stroke chain of careJournal of Interprofessional Care10.1080/13561820.2024.234407538:4(664-674)Online publication date: 8-May-2024
https://doi.org/10.1080/13561820.2024.2344075
Nishide SNishida T(2024)Using Word Games as Facilitator to Awareness Raising Communication in Public SpacesCollaboration Technologies and Social Computing10.1007/978-3-031-67998-8_15(212-219)Online publication date: 20-Aug-2024
https://doi.org/10.1007/978-3-031-67998-8_15
Correia AGrover ASchneider DPimentel AChaves Rde Almeida MFonseca B(2023)Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine InteractionApplied Sciences10.3390/app1304219813:4(2198)Online publication date: 8-Feb-2023
https://doi.org/10.3390/app13042198
Berger MDey DBarati BPfleging BBernhaupt R(2023)Designing for Collaborative Non-Driving Related Activities in Future Cars: Fairness and Team PerformanceProceedings of the ACM on Human-Computer Interaction10.1145/36042497:MHCI(1-28)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604249
Cibulski LMay TSchmidt JKohlhammer J(2023)COMPO*SED: Composite Parallel Coordinates for Co-Dependent Multi-Attribute ChoicesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.318089929:10(4047-4061)Online publication date: 1-Oct-2023
https://doi.org/10.1109/TVCG.2022.3180899
Zhu BBao THan RCui HHan JLiu LPeng T(2023)An effective knowledge graph entity alignment model based on multiple informationNeural Networks10.1016/j.neunet.2023.02.029162:C(83-98)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1016/j.neunet.2023.02.029
Berger MVerstegen RBarati Bvan Essen HBernhaupt R(2023)Collaborative TV Control: Towards Co-experience and Social ConnectednessHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42286-7_20(369-392)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-42286-7_20
Letter MGeorge CWolf K(2023)A Survey of Computer-Supported Remote Collaboration on Physical ObjectsHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42286-7_19(346-368)Online publication date: 25-Aug-2023
https://doi.org/10.1007/978-3-031-42286-7_19
Avdullahu AHerrmann TRummel N(2023)Requirements Analysis to Support Equal Participation in Hybrid Collaboration Settings in Higher EducationLearning and Collaboration Technologies10.1007/978-3-031-34550-0_30(411-430)Online publication date: 9-Jun-2023
https://doi.org/10.1007/978-3-031-34550-0_30
Hasan MMuna F(2022)Technology Trends and Cyber Security in BangladeshInternational Journal of Technology Diffusion10.4018/IJTD.30438113:1(1-17)Online publication date: 8-Jul-2022
https://dl.acm.org/doi/10.4018/IJTD.304381
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten