research-article

Structuring dimensions for collaborative systems evaluation

Authors:

Valeria Herskovic,

Sergio F. Ochoa,

Jose A. PinoAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 44, Issue 2

Article No.: 8, Pages 1 - 28

https://doi.org/10.1145/2089125.2089128

Published: 05 March 2008 Publication History

Abstract

Collaborative systems evaluation is always necessary to determine the impact a solution will have on the individuals, groups, and the organization. Several methods of evaluation have been proposed. These methods comprise a variety of approaches with various goals. Thus, the need for a strategy to select the most appropriate method for a specific case is clear. This research work presents a detailed framework to evaluate collaborative systems according to given variables and performance levels. The proposal assumes that evaluation is an evolving process during the system lifecycle. Therefore, the framework, illustrated with two examples, is complemented with a collection of guidelines to evaluate collaborative systems according to product development status.

References

[1]

Antunes, P. and Costa, C. 2003. Perceived value: A low-cost approach to evaluate meetingware. In Proceedings of CRIWG'03. Lecture Notes in Computer Science, vol. 2806, 109--125.

[2]

Antunes, P., Ferreira, A., and Pino, J. 2006. Analyzing shared workspace design with human-performance models. In Proceedings of CRIWG'06. Lecture Notes in Computer Science, vol. 4154, 62--77.

Digital Library

[3]

Antunes, P., Ramires, J., and Respício, A. 2006. Addressing the conflicting dimension of groupware: A case study in software requirements validation. Comput. Informatics 25, 523--546.

[4]

Araujo, R., Santoro, F., and Borges, M. 2002. The CSCW lab for groupware evaluation. In Proceedings of CRIWG'02. Lecture Notes in Computer Science, vol. 2440, 222--231.

Digital Library

[5]

Baecker, R. M., Grudin, J., Buxton, W., and Greenberg, S., Eds. 1995. Human-computer Interaction: Toward the Year 2000. Morgan Kaufmann, San Francisco, CA, 1995.

Digital Library

[6]

Baeza-Yates, R. and Pino, J. 1997. A first step to formally evaluate collaborative work. In Proceedings of the ACM International Conference on Supporting GroupWork (GROUP ‘97). 55--60.

Digital Library

[7]

Baeza-Yates, R. and Pino, J. 2006. Towards formal evaluation of collaborative work and its application to information retrieval. Info. Res. 11, 4.

[8]

Baker, K., Greenberg, S., and Gutwin, C. 2001. Heuristic evaluation of groupware based on the mechanics of collaboration. In Proceedings of the 8th IFIP International Conference on Engineering For Human-Computer interaction. Lecture Notes in Computer Science, vol. 2254, 123--140.

Digital Library

[9]

Baker, K., Greenberg, S., and Gutwin, C. 2002. Empirical development of a heuristic evaluation methodology for shared workspace groupware. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. 96--105.

Digital Library

[10]

Bias, R. 1991. Interface-walkthroughs: Efficient collaborative testing. IEEE Softw. 8, 5, 94--95.

Digital Library

[11]

Bias, R. 1994. The pluralistic usability walkthrough: coordinated empathies. Usability Inspection Methods. J. Nielsen and R. Mack, Eds., John Wiley & Sons, New York, 63--76.

Digital Library

[12]

Briggs, R., Adikns, M., Mittleman, D., Kruse, J., Miller, S., and Nunamaker, J. 1998. A technology transition model derived from field investigation of GSS use aboard the U.S.S. CORONADO. J. Manage. Info. Syst. 15, 3, 151--195.

Digital Library

[13]

Briggs, R., Qureshi, S., and Reinig, B. 2004. Satisfaction attainment theory as a model for value creation. In Proceedings of the 37th Annual Hawaii International Conference on Systems Sciences, IEEE Computer Society Press.

Digital Library

[14]

Card, S., Moran, T., and Newell, A. 1980. The keystroke-level model for user performance time with interactive systems. Comm. ACM 23, 7, 396--410.

Digital Library

[15]

Carroll, J. 2000. Making use: Scenario-Based Design of Human-Computer Interactions. The MIT Press, Cambridge, MA.

Digital Library

[16]

Cockton, G. and Woolrych, A. 2002. Sale must end: should discount methods be cleared off HCI’s shelves&quest; Interactions 9, 5, 13--18.

Digital Library

[17]

Conevrtino, G., Neale, D., Hobby, L., Carroll, J., and Rosson, M. 2004. A laboratory method for studying activity awareness. In Proceedings of the 3rd Nordic Conference on Human-Computer Interaction. 313--322.

Digital Library

[18]

Damianos, L., Hirschman, L., Kozierok, R., Kutrz, J., Greenberg, A., Walls, K., Laskowski, S., and Scholtz, J. 1999. Evaluation for collaborative systems. ACM Comput. Surv. 31, 2, 15.

Digital Library

[19]

Desanctis, G., Snyder, J., and Poole, M. 1994. The meaning of the interface: a functional and holistic evaluation of a meeting software system. Decis. Supp. Syst. 11, 319--335.

Digital Library

[20]

Ereback, A. and Höök, K. 1994. Using cognitive walkthrough for evaluating a CSCW application. In Proceedings of the Conference Companion on Human Factors in Computing Systems, 91--92.

Digital Library

[21]

Ferreira, A., Antunes, P., and Pino, J. 2009. Evaluating shared workspace performance using human information processing models. Info. Res. 14, 1, 388.

[22]

Fjermestad, J. and Hiltz, S. 1999. An assessment of group support systems experimental research: methodology and results. J. Manag. Info. Syst. 15, 3, 7--149.

Digital Library

[23]

Greenberg, S. and Buxton, B. 2008. Usability evaluation considered harmful (some of the time). In Proceedings of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems, 111--120.

Digital Library

[24]

Gutwin, C. and Greenberg, S. 1999. The effects of workspace awareness support on the usability of real-time distributed groupware. ACM Trans. Comput.-Human Interact. 6, 3, 243--281.

Digital Library

[25]

Gutwin, C. and Greenberg, S. 2000. The mechanics of collaboration: Developing low cost usability evaluation methods for shared workspaces. In Proceedings of the IEEE International Workshops on Enabling Technologies Infrastructures for Collaborative Enterprises, 98--103.

Digital Library

[26]

Gutwin, C., Roseman, M., and Greenberg, S. 1996. A usability study of awareness widgets in a shared workspace groupware system. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, 258--267.

Digital Library

[27]

GUY, E. 2005. “…real, concrete facts about what works…”: integrating evaluation and design through patterns. In Proceedings of the International ACM SIGGROUP Conference on Supporting Group Work, 99--108.

Digital Library

[28]

Haynes, S., Purao, S., and Skattebo, A. 2004. Situating evaluation in scenarios of use. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, 92--101.

Digital Library

[29]

Herskovic, V., Pino, J. A., Ochoa, S. F., and Antunes, P. 2007. Evaluation methods for groupware systems. In Proceedings of CRIWG, Lecture Notes in Computer Science, vol. 4715, 328--336.

Digital Library

[30]

Hix, D. and Hartson, H. R. 1993. Developing User Interfaces: Ensuring Usability Through Product and Process. John Wiley & Sons, Inc., New York, NY.

Digital Library

[31]

Huang, J. P. H. 2005. A conceptual framework for understanding collaborative systems evaluation. In Proceedings of the 14th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprise. 215--220.

Digital Library

[32]

Hughes, J., King, V., Rodden, T., and Andersen, H. 1994a. Moving out from the control room: ethnography in system design. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. 429--439.

Digital Library

[33]

Hughes, J., Sharrock, W., Rodden, T., O'Brien, J., Rouncefield, M., and Calvey, D. 1994b. Field Studies and CSCW. Lancaster University, Lancaster, U.K.

[34]

Humphries, W., Neale, D., Mccrickard, D., and Carroll, J. 2004. Laboratory simulation methods for studying complex collaborative tasks. In Proceedings of the 48th Annual Meeting Human Factors and Ergonomics Society, 2451--2455.

[35]

Inkpen, K., Mandryk, R., Dimicco, J., and Scott, S. 2004. Methodology for evaluating collaboration behaviour in co-located environments. In Proceedings of the ACM Conference on Computer Supported Cooperative Work.

[36]

Kieras, D. and Santoro, T. 2004. Computational GOMS modeling of a complex team task: lessons learned. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 97--104.

Digital Library

[37]

Mcgrath, J. 1984. Groups: Interaction and Performance. Prentice-Hall, Englewood Cliffs, NJ.

[38]

Neale, D. and Carroll, J. 1999. Multi-faceted evaluation for complex, distributed activities. In Proceedings of the Conference on Computer Support For Collaborative Learning. 425--433.

Digital Library

[39]

Neale, D., Carroll, J., and Rosson, M. 2004. Evaluating computer-supported cooperative work: models and frameworks. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, 112--121.

Digital Library

[40]

Newell, A. 1990. Unified Theories of Cognition. Harvard University Press, Cambridge, MA.

Digital Library

[41]

Nielsen, J. 1989. Usability engineering at a discount. In Designing and Using Human-Computer Interfaces and Knowledge Based Systems, G. Salvendy and M. Smith, Eds., Elsevier Science Publishers, Amsterdam, 394--401.

Digital Library

[42]

Nielsen, J. 1994. Usability inspection methods. In Proceedings of the Conference on Human Factors in Computing Systems, 413--414.

Digital Library

[43]

Nielsen, J. and Molich, R. 1990. Heuristic evaluation of user interfaces. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 249--256.

Digital Library

[44]

Ochoa, S., Pino, J., Bravo, G., Dujovne, N., and Neyem, A. 2008. Mobile shared workspaces to support construction inspection activities. In Collaborative Decision Making: Perspectives and Challenges, P. Zarate, J. Belaud, G. Camileri, and F. Ravat, Eds., IOS Press, Amsterdam, 211--220.

Digital Library

[45]

Pidd, M. 1996. Tools for Thinking. J. Wiley & Sons, Chichester.

[46]

Pinelle, D. and Gutwin, C. 2000. A review of groupware evaluations. In Proceedings of the 9th IEEE WETICE Infrastructure for Collaborative Enterprises, 86--91.

Digital Library

[47]

Pinelle, D. and Gutwin, C. 2002. Groupware walkthrough: adding context to groupware usability evaluation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 455--462.

Digital Library

[48]

Pinelle, D. and Gutwin, C. 2008. Evaluating teamwork support in tabletop groupware applications using collaboration usability analysis. Pers. Ubiq. Comput. 12, 3, 237--254.

Digital Library

[49]

Pinelle, D., Gutwin, C., and Greenberg, S. 2003. Task analysis for groupware usability evaluation: modeling shared-workspace tasks with the mechanics of collaboration. ACM Trans. Comput.-Human Interact. 10, 4, 281--311.

Digital Library

[50]

Pinsonneault, A. and Kraemer, K. 1989. The impact of technological support on groups: an assessment of the empirical research. Decis. Supp. Syst. 5, 3, 197--216.

[51]

Plowman, L., Rogers, Y., and Ramage, M. 1995. What are workplace studies for&quest; In Proceedings of the 4th European Conference on Computer-Supported Cooperative Work, 309--324.

Digital Library

[52]

Polson, P. G., Lewis, C., Rieman, J., and Wharton, C. 1992. Cognitive walkthroughs: a method for theory- based evaluation of user interfaces. Int. J. Man-Mach. Stud. 36, 5, 741--773.

Digital Library

[53]

Rasmussen, J. and Jensen, A. 1974. Mental procedures in real-life tasks : a case-study of electronic trouble shooting. Ergonomics 17, 293--307.

[54]

Reason, J. 2008. The Human Contribution: Unsafe Acts, Accidents and Heroic Recoveries. Ashgate, Surrey, UK.

[55]

Ross, S., Ramage, M., and Rogers, Y. 1995. PETRA: participatory evaluation through redesign and analysis. Interact. Comput. 7, 4, 335--360.

[56]

Rowley, D. and Rhoades, D. 1992. The cognitive jogthrough: a fast-paced user interface evaluation procedure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 389--395.

Digital Library

[57]

Ruhleder, K. and Jordan, B. 1998. Video-based interaction analysis (VBIA) in distributed settings: a tool for analyzing multiple-site, technology-supported interactions. In Proceedings of the Participatory Design Conference, 195--196.

[58]

Scriven, M. 1967. The methodology of evaluation. In Perspectives of Curriculum Evaluation, R. Tyler, R. Gagne, and M. Scriven, Eds., Rand McNally, Chicago, 39--83.

[59]

Sonnenwald, D., Maglaughlin, K., and Whitton, M. 2001. Using innovation diffusion theory to guide collaboration technology evaluation: work in progress. In Proceedings of the IEEE International Workshop on Enabling Technologies, 114--119.

Digital Library

[60]

Steves, M., Morse, E., Gutwin, C., and Greenberg, S. 2001. A comparison of usage evaluation and inspection methods for assessing groupware usability. In Proceedings of the International ACM SIGGROUP Conference on Supporting Group Work, 125--134.

Digital Library

[61]

Stiemerling, O. and Cremers, A. 1998. The use of cooperation scenarios in the design and evaluation of a CSCW system. IEEE Trans. Softw. Eng. 24, 12, 1171--1181.

Digital Library

[62]

Suchman, L. 1987. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge University Press, Cambridge, U.K.

Digital Library

[63]

Tang, J. 1991. Findings from observational studies of collaborative work. Intern. J. Man-Machine Stud. 34, 2, 143--160.

Digital Library

[64]

Twidale, M., Randall, D., and Bentley, R. 1994. Situated evaluation for cooperative systems. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, 441--452.

Digital Library

[65]

Urquijo, S., Scrivener, S., and Palmen, H. 1993. The use of breakdown analysis in synchronous CSCW system design. In Proceedings of the 3rd European Conference on Computer Supported Cooperative Work—ECSCW 93, 289--302.

Digital Library

[66]

Van Der Veer, G. 2000. Task based groupware design: putting theory into practice. In Proceedings of the Symposium on Designing Interactive Systems. 326--337.

Digital Library

[67]

Van Der Veer, G., Lenting, B., and Bergevoet, B. 1996. Gta: Groupware task analysis—modeling complexity. Acta Psychologica 91, 3, 297--322.

[68]

Veld, M. A. A. H. I. T., Andriessen, J. H. E., and Verburg, R. M. 2003. E-MAGINE: The development of an evaluation method to assess groupware applications. In Proceedings of the 12th International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises, 153--158.

Digital Library

[69]

Vizcaíno, A., Martinez, M., Aranda, G., and Piattini, M. 2005. Evaluating collaborative applications from a knowledge management approach. In Proceedings of the 14th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprise (WETICE'05), 221--225.

Digital Library

[70]

Wharton, C., Rieman, J., Lewis, C., and Polson, P. 1994. The cognitive walkthrough method: a practictioner's guide. In Usability Inspection Methods, J. Nielsen and R. Mack, Eds., John Wiley & Sons, New York, 105--140.

Digital Library

[71]

Yourdon, E. 1978. Structured Walkthroughs. Yourdon Inc, New York.

Digital Library

Cited By

Mantau MBenitti F(2024)The awareness assessment model: measuring awareness and collaboration support over participant’s perspectiveUniversal Access in the Information Society10.1007/s10209-024-01110-5Online publication date: 14-Mar-2024
https://doi.org/10.1007/s10209-024-01110-5
Haque AIslam AMikalef P(2023)Explainable Artificial Intelligence (XAI) from a user perspective: A synthesis of prior literature and problematizing avenues for future researchTechnological Forecasting and Social Change10.1016/j.techfore.2022.122120186(122120)Online publication date: Jan-2023
https://doi.org/10.1016/j.techfore.2022.122120
Hvannberg EHalldorsdottir GRudinsky J(2022)The Usefulness of Method-Resources for Evaluating a Collaborative Training SimulatorHuman-Computer Interaction – INTERACT 201510.1007/978-3-319-22668-2_9(98-105)Online publication date: 10-Mar-2022
https://dl.acm.org/doi/10.1007/978-3-319-22668-2_9
Show More Cited By

Index Terms

Structuring dimensions for collaborative systems evaluation
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing design and evaluation methods
    2. Collaborative and social computing theory, concepts and paradigms
2. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Project and people management
        Systems development

Recommendations

Caracterização das adaptações em métodos de avaliação para aplicações colaborativas
IHC '12: Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems

Although the development and use of collaborative systems has increased lately, evaluate them is not a simple task. Most of the HCI evaluation methods that are consolidated are aimed at single-user systems and require adaptation in order to be applied ...
Structuring the knowledge-based systems evaluation process
IIS '97: Proceedings of the 1997 IASTED International Conference on Intelligent Information Systems (IIS '97)

Knowledge-based systems (KBS) evaluation is still an obscure issue for practitioners who need to plan, control and conduct the evaluation process. This situation can be partly put down to the fact that there are no general models or frameworks for KBS ...
Uma análise comparativa dos métodos de avaliação de sistemas colaborativos fundamentados na engenharia semiótica
IHC '13: Proceedings of the 12th Brazilian Symposium on Human Factors in Computing Systems

Collaborative systems are growing more and more popular. However, evaluating this type of systems is still a difficult task. Many evaluation methods have been proposed to deal with this challenge, but most of them are not yet consolidated. Semiotic ...

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 44, Issue 2

February 2012

132 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/2089125

Issue’s Table of Contents

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Accepted: 01 August 2010

Revised: 01 June 2010

Received: 01 February 2010

Published: 05 March 2008

Published in CSUR Volume 44, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Fondo Nacional de Desarrollo Científico y Tecnológico

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

49
Total Citations
View Citations
1,800
Total Downloads

Downloads (Last 12 months)52
Downloads (Last 6 weeks)3

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mantau MBenitti F(2024)The awareness assessment model: measuring awareness and collaboration support over participant’s perspectiveUniversal Access in the Information Society10.1007/s10209-024-01110-5Online publication date: 14-Mar-2024
https://doi.org/10.1007/s10209-024-01110-5
Haque AIslam AMikalef P(2023)Explainable Artificial Intelligence (XAI) from a user perspective: A synthesis of prior literature and problematizing avenues for future researchTechnological Forecasting and Social Change10.1016/j.techfore.2022.122120186(122120)Online publication date: Jan-2023
https://doi.org/10.1016/j.techfore.2022.122120
Hvannberg EHalldorsdottir GRudinsky J(2022)The Usefulness of Method-Resources for Evaluating a Collaborative Training SimulatorHuman-Computer Interaction – INTERACT 201510.1007/978-3-319-22668-2_9(98-105)Online publication date: 10-Mar-2022
https://dl.acm.org/doi/10.1007/978-3-319-22668-2_9
Li JLin DWang YXu GDing C(2021)Towards a Reliable Evaluation of Local Interpretation MethodsApplied Sciences10.3390/app1106273211:6(2732)Online publication date: 18-Mar-2021
https://doi.org/10.3390/app11062732
Alrashedi AAbbod M(2020)The Effect of Using Artificial Intelligence on Performance of Appraisal System: A Case Study for University of Jeddah Staff in Saudi ArabiaIntelligent Systems and Applications10.1007/978-3-030-55180-3_11(145-154)Online publication date: 25-Aug-2020
https://doi.org/10.1007/978-3-030-55180-3_11
Chounta I(2020)Collaborative Learning and Patterns of PracticeEncyclopedia of Education and Information Technologies10.1007/978-3-030-10576-1_83(310-323)Online publication date: 14-Jun-2020
https://doi.org/10.1007/978-3-030-10576-1_83
Yuan XPino J(2019)Cyber Behaviors in Seeking Health InformationAdvanced Methodologies and Technologies in Medicine and Healthcare10.4018/978-1-5225-7489-7.ch016(198-209)Online publication date: 2019
https://doi.org/10.4018/978-1-5225-7489-7.ch016
Baloian NFrez JPino JPeñafiel SZurita GAbarca A(2019)Technology Support for Collaborative Preparation of Emergency PlansSensors10.3390/s1922504019:22(5040)Online publication date: 19-Nov-2019
https://doi.org/10.3390/s19225040
Frías SMezura-Godoy CBenítez-Guerrero ESantana-Mancilla PValderrama E(2019)FrUtEGProceedings of the IX Latin American Conference on Human Computer Interaction10.1145/3358961.3358965(1-8)Online publication date: 30-Sep-2019
https://dl.acm.org/doi/10.1145/3358961.3358965
Chounta I(2019)Collaborative Learning and Patterns of PracticeEncyclopedia of Education and Information Technologies10.1007/978-3-319-60013-0_83-1(1-14)Online publication date: 14-Aug-2019
https://doi.org/10.1007/978-3-319-60013-0_83-1
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents