research-article

Evaluation in participatory design: a literature survey

Authors:

Christian Dindler,

Ole Sejer IversenAuthors Info & Claims

PDC '16: Proceedings of the 14th Participatory Design Conference: Full papers - Volume 1

Pages 151 - 160

https://doi.org/10.1145/2940299.2940303

Published: 15 August 2016 Publication History

Abstract

This paper focuses on evaluation in Participatory Design (PD), and especially upon how the central aims of mutual learning, empowerment, democracy and workplace quality have been assessed. We surveyed all Participatory Design Conference papers (1990-2014) and papers from special journal issues on PD, focusing on systematic, explicit evaluations. The survey resulted in 143 papers of which 66 were deemed relevant. Of these, 17 papers deal with evaluation of the above mentioned aims. Based on evaluation theory, we propose seven key questions through which to characterize evaluations in PD and analyze the 17 papers. Our analysis reveals that formal evaluations of PD's aims are rare; generally lack details on methods; are researcher- and not participant-led, and that a corpus of work around evaluations needs to be developed. We suggest more explicit, systematic evaluations of PD's central aims to enhance accountability, learning and knowledge building, and to strengthen PD internally and externally.

References

[1]

Beck, E., Obrist, M, Bernhaupt, R. and Tscheligi, M. 2008. Instant card technique: how and why to apply in user-centered design. In Proc. of PDC '08. IN, USA, p. 162--165.

Digital Library

[2]

van den Besselaar, P. 1998. Democracy & technological change: Limits to steering. In Proc. of PDC'98, Seattle, WA. USA, p. 1--10.

[3]

Reich, B. H., Gemino, A. and Sauer, C. 2008. Modeling the knowledge perspective of IT projects. Project Management Journal, 39(S1), S4-S14. DO=10.1002/pmj.20056

[4]

Bossen, C., Dindler, C. and Iversen, O. S. 2010. User gains and PD aims: assessment from a participatory design project. In Proc. of PDC '10. ACM, New York, NY, USA, p. 141--150. DOI=http://dx.doi.org/10.1145/1900441.1900461

Digital Library

[5]

Bossen, C., Dindler, C. and Iversen, O. S. 2012. Impediments to user gains: experiences from a critical participatory design project. In Proc. Of PDC'12, Vol. 1. ACM, New York, NY, USA, p. 31--40. DOI=http://dx.doi.org/10.1145/2347635.2347641

Digital Library

[6]

Braa, K. 1992. Influencing System Quality by Using Decision Diaries in Prototyping Projects, in Proc. of PDC'92, Palo Alto, CA. USA p. 163--170.

[7]

Brandon, P. R. and Ah Sam, A. L. 2014. Program Evaluation. In: The Oxford Handbook of Qualitative Research, Patricia Leavy (ed), p. 471--497.

[8]

Bratteteig, T. and Wagner, I. 2014. Design decisions and the sharing of power in PD. In Proc. of PDC '14, Vol. 2. ACM, New York, NY, USA, p. 29--32. DOI=http://dx.doi.org/10.1145/2662155.2662192

Digital Library

[9]

Clement, A., Griffiths, M. and van den Besselaar, P. 1992. Participatory Design Projects: A retrospective Look. Proc. of PDC'92, Palo Alto, CA. USA, 81--89.

[10]

Costantino, T. LeMay, S. Vizard, L., Moore, H., Renton, D. Gornall, S. and Strang, I. 2014. Participatory design of public library e-services. Proc. of PDC'14, Windhoek, Namibia, p. 133--136. DOI=http://dx.doi.org/10.1145/2662155.2662232

Digital Library

[11]

Dahler-Larsen, P. 2011, The Evaluation Society. Stanford, Stanford University Press, 2011.

[12]

Ehn, P. 1990. Work-Oriented Design of Computer Artifacts. L. Erlbaum Assoc. Inc., Hillsdale, NJ, USA.

Digital Library

[13]

Frauenberger, C. Good, J. Fitzpatrick, G. and Iversen, O. S. 2015. In pursuit of rigour and accountability in participatory design. In: International Journal of Human Computer Studies, 74, p. 93--108.

Digital Library

[14]

Garde, J. A. and van der Voort, M. C. 2014. Participants' view on personal gains and PD process. In Proc. of PDC '14, Vol. 2. ACM, New York, NY, USA, p. 79--82. DOI=http://dx.doi.org/10.1145/2662155.2662194

Digital Library

[15]

Gerrard, V. and Sosa, R. 2014. Examining participation. In Proc. of PDC '14, Vol. 1. ACM, New York, NY, USA, p. 111--120. DOI=http://dx.doi.org/10.1145/2661435.2661451

Digital Library

[16]

Greene, J. C. 2004. Evaluation, Democracy and Social Change. In The Sage Handbook of Evaluation, Ian F. Shaw, Jennifer C. Greene & Melvin M Mark (eds), p. 118--140.

[17]

Guba, E. G. and Lincoln, Y. S. 1989 "Fourth Generation Evaluation", Newbury Park, Sage.

[18]

Hallberg, N., Pilemalm, S., Jägare, D., Irestig, M. and Timpka, T. 2000. Quality function Deployment (QFD) extended Future Workshop: An Approach for Effective and Enjoyable User Participation, Proc. Of. PDC'00, New York, New York, USA, p. 86--95.

[19]

Hemmings, T., Clarke, K. Rouncefield, M., Crabtree, A. Rodden, T. 2002. Probing the Probes, in Proc. of PDC'02, Malmö, Sweden, p. 42--50.

[20]

Hertzum, M. and Simonsen, J. 2010. Effects-driven IT development: an instrument for supporting sustained participatory design. In Proc. of PDC '10. ACM, New York, NY, USA, p. 61--70. DOI=http://dx.doi.org/10.1145/1900441.1900451

Digital Library

[21]

Hirschheim, R. A. 1983. Assessing participative systems design: Some conclusions from an exploratory study. Information & Management. 1983;6(6):317--27. DOI=http://dx.doi.org/10.1016/0378-7206(83)90040-X

[22]

Hornecker, E. Eden, H. and Scharff, E. 2002. In MY situation I would dislike THAAAT!--- Role Play as Assessment Method for Tools Supporting Participatory Planning, in Proc. of PDC'02, Malmö, Sweden, p. 243--247.

[23]

Hänninen, R. and Blom, J. 2012. Every breath you take: use of sensitizing methods in the design of air quality services. In Proc. of PDC'12, Vol. 2. ACM, New York, NY, USA, p. 17--20. DOI=http://dx.doi.org/10.1145/2348144.2348151

Digital Library

[24]

Kensing, F. and Blomberg, J. 1998. Participatory Design: Issues and Concerns, in Journal of CSCW, Kluwer Academic Publishers 7: p. 167--185.

Digital Library

[25]

Kensing, F., Simonsen, J. and Bødker, K. 1996. MUST - a Method for Participatory Design, In Proc. of PDC'96, Cambridge, MA, p. 129--140.

[26]

Madaus, G. F. and Stufflebeam, D. L. 2000. Program Evaluation: A Historical Overview. In Daniel L. Stufflebeam, George F. Madaus, Thomas Kellaghan (eds). Evaluation Models, Kluwer, pp3--18.

[27]

Mambrey, P., Mark, G. and Pankoke-Babatz, U. 1996. In Proc. of PDC'96, Cambridge, MA, USA, p. 251--259.

[28]

Mark, M. M. Greene, J. C. and Shaw, I. E. 2004. Introduction: The Evaluation of Policies, Programs, and Practices. In: The Sage Handbook of Evaluation, Melvin M. Mark, Jennifer C. Greene & Ian E. Shaw (eds), p. 1--30.

[29]

Merkel, C. B., Xiao, L. Farooq, U. Ganoe, G. H., Lee, R., Carroll, J. M. and Rosson, M. B. 2004. Participatory design in community computing contexts: tales from the field. In Proc. of PDC 04, Vol. 1. ACM, New York, NY, USA, p. 1--10. DOI=http://dx.doi.org/10.1145/1011870.1011872

Digital Library

[30]

Muller, M. and Druin, A. 2010. Participatory Design: The Third Space in Human-Computer Interaction, in The Human-Interaction Handbook. Hillsdale, NJ: Lawrence Erlbaum Associates.

[31]

Kapuire, G. K. Winschiers-Theophilus, H. and Blake, E. 2015. An insider perspective on community gains: A subjective account of Namibian rural communities' perception of a long-term participatory design project in International Journal of Human-Computer Studies, Volume 74, 124--143.

Digital Library

[32]

Patton, M. Q. 1997. Utilization-Focused Evaluation. Thousand Oaks, CA. Sage.

[33]

Pollack, J. 2007. The changing paradigms of project management. International Journal of Project Management, 25, 266--274, DOI=10.1016/j.ijproman.2006.08.002

[34]

Power, M. 1997. The Audit Socity. Oxford, Oxford University Press

[35]

Robertson, T. and Wagner, I. 2012. Ethics: engagement, representation and politics-in-action. in Routledge International Handbook of Participatory Design. Routledge, Taylor & Francis Ltd.

[36]

Shadish, W. S., Cook, T. D. and Leviton, L. C. 1991. "Foundations of Program Evaluation", Newbury Park, Sage.

[37]

Shapiro, D. 2005. Participatory design: the will to succeed. In Proc. of Critical computing (CC '05), ACM, New York, NY, USA, p. 29--38. DOI=http://dx.doi.org/10.1145/1094562.1094567

Digital Library

[38]

Simonsen, J. and Hertzum, M. 2008. Participative design and the challenges of large-scale systems: extending the iterative PD approach. In Proc. of PDC '08. Indianapolis, IN, USA, p. 1--10.

Digital Library

[39]

Simonsen, J. and Hertzum, M. 2012. Real-use evaluation of effects: emergency departments aiming for 'Warm Hands'. In Proc. of PDC'12), Vol. 2. ACM, New York, NY, USA, p. 69--72. DOI=http://dx.doi.org/10.1145/2348144.2348166

Digital Library

[40]

Simonsen, J. and Robertson, T. 2012. Routledge International Handbook of Participatory Design. Routledge, Taylor & Francis Ltd.

[41]

Taxén, G. 2004. Introducing participatory design in museums. In Proc. of PDC 04, Vol. 1. ACM, New York, NY, USA, p. 204--213. DOI=http://dx.doi.org/10.1145/1011870.1011894

Digital Library

[42]

Thoresen, K. 1990. Prototyping organizational change. In Proc. of PDC'90, p. 22--35.

[43]

Timpka, T. and Sjöberg, C. 1994. Voices in Design: The Dynamics of Participatory Information System Design. In Proc. of PDC'94, Palo Alto, CA, USA p. 75--85.

[44]

Whittle, J. 2014. How much participation is enough? A comparison of six participatory design projects in terms of outcomes. In Proc. of PDC '14, Vol. 1. ACM, New York, NY, USA, p. 121--130. DOI=http://dx.doi.org/10.1145/2661435.2661445

Digital Library

[45]

Winchiers-Theophilus, H. 2006. The Challenges of Participatory Design in a Intercultural Context: Designing for Usability in Namibia, Proc. PDC '06, ACM, NY, p. 73--76.

Cited By

Bamford JMoschini E(2025)Co-creation of connected and relational learning spaces: a digital and gamified approach in higher educationHigher Education Research & Development10.1080/07294360.2024.242944344:1(205-221)Online publication date: 29-Jan-2025
https://doi.org/10.1080/07294360.2024.2429443
Lepe-Salazar FEscobedo LNakajima T(2025)Novice-friendly probes for the gathering and analysis of requirements and subsequent design of softwareInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103405195:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103405
Dorton SStanley J(2024)Minding the Gap: Tools for Trust Engineering of Artificial IntelligenceErgonomics in Design: The Quarterly of Human Factors Applications10.1177/10648046241249903Online publication date: 7-May-2024
https://doi.org/10.1177/10648046241249903
Show More Cited By

Index Terms

Evaluation in participatory design: a literature survey
1. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods
      1. Participatory design

Recommendations

What is a participatory design result?
PDC '16: Proceedings of the 14th Participatory Design Conference: Full papers - Volume 1

In this paper we discuss what the result of a Participatory Design (PD) process is and how it can be described and evaluated. We look at several PD projects and discuss if they have a participatory result and how we know that it is participatory. We ...
Applying participatory design theory to designing evaluation methods
CHI EA '12: CHI '12 Extended Abstracts on Human Factors in Computing Systems

System evaluators face several challenges in designing evaluation methods, including measurement and relevance, context, establishing common ground with users, and eliciting users' tacit knowledge. To address these challenges, we propose applying ...
The role of design fiction in participatory design processes
NordiCHI '18: Proceedings of the 10th Nordic Conference on Human-Computer Interaction

Participatory design is in essence very malleable as any design technique could lend itself to it, as long as users and stakeholders are involved. Design fictions however, have more often been used as either a vehicle for critical designs, or as a sheer ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PDC '16: Proceedings of the 14th Participatory Design Conference: Full papers - Volume 1

August 2016

192 pages

ISBN:9781450340465

DOI:10.1145/2940299

Editors:
Claus Bossen
Aarhus Universitet, Denmark
,
Rachel Charlotte Smith
Aarhus Universitet, Denmark
,
Anne Marie Kanstrup
E-learning lab, Department of Communication and Psychology, The Faculty of Humanities, Aalborg University
,
Janet McDonnell
University of the Arts London, UK
,
Maurizio Teli
Madeira Interactive Technologies Institute, Funchal, Portugal
,
Keld Bødker
Roskilde Universitetscenter, Denmark

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Aarhus University: Aarhus University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 August 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

PDC '16

Sponsor:

Aarhus University

PDC '16: The 14th Participatory Design Conference

August 15 - 19, 2016

Aarhus, Denmark

Acceptance Rates

Overall Acceptance Rate 49 of 289 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

61
Total Citations
View Citations
2,991
Total Downloads

Downloads (Last 12 months)727
Downloads (Last 6 weeks)77

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bamford JMoschini E(2025)Co-creation of connected and relational learning spaces: a digital and gamified approach in higher educationHigher Education Research & Development10.1080/07294360.2024.242944344:1(205-221)Online publication date: 29-Jan-2025
https://doi.org/10.1080/07294360.2024.2429443
Lepe-Salazar FEscobedo LNakajima T(2025)Novice-friendly probes for the gathering and analysis of requirements and subsequent design of softwareInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103405195:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103405
Dorton SStanley J(2024)Minding the Gap: Tools for Trust Engineering of Artificial IntelligenceErgonomics in Design: The Quarterly of Human Factors Applications10.1177/10648046241249903Online publication date: 7-May-2024
https://doi.org/10.1177/10648046241249903
Lowy RMagiawala KMittal SHall KRoberts JKim J(2024)Research-Education Partnerships: A Co-Design Classroom for College Students with Intellectual and Developmental DisabilitiesProceedings of the ACM on Human-Computer Interaction10.1145/36870508:CSCW2(1-26)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687050
Nguyen QJaspaert EHarthum B(2024)Talk the Talk, Walk the Walk: A Case Study on Systematic Evaluation in Participatory DesignProceedings of the Participatory Design Conference 2024: Full Papers - Volume 110.1145/3666094.3666103(133-144)Online publication date: 11-Aug-2024
https://dl.acm.org/doi/10.1145/3666094.3666103
Dindler CIversen OKanstrup KVan Mechelen MWagner M(2024)Participatory Design Sprints - Employing PD Principles in a Condensed FormatProceedings of the Participatory Design Conference 2024: Exploratory Papers and Workshops - Volume 210.1145/3661455.3669876(93-97)Online publication date: 11-Aug-2024
https://dl.acm.org/doi/10.1145/3661455.3669876
Nguyen QJaspaert EMurtinger MKriglstein STscheligi M(2024)CoLEBricks: Co-Designing Virtual Reality Scenarios with Generative Building BlocksProceedings of the 2024 International Conference on Advanced Visual Interfaces10.1145/3656650.3656686(1-5)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3656650.3656686
Nguyen QWindisch AKriglstein S(2024)Between Two Worlds: Analysing the Effects of Immersive and Non-Immersive Prototyping for Participatory DesignProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660715(2198-2212)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660715
DiSalvo CRothschild ASchenck LShapiro BDiSalvo B(2024)When Workers Want to Say No: A View into Critical Consciousness and Workplace Democracy in Data WorkProceedings of the ACM on Human-Computer Interaction10.1145/36374338:CSCW1(1-24)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637433
Wilner TAdavi KMandava SBhimdiwala AFrluckaj HTurns JArif A(2024)From Concept to Community: Unpacking the Work of Designing Educational and Activist ToolkitsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642681(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642681
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten