Community-based evaluation in online communities: Considering the materiality of (making) “best practice”

The idea of “best practice” is very much built into information systems and the ways in which they organise and structure work. The purpose of this paper is to examine how “best practice” may be identified (produced) through a community-based evaluation process as opposed to traditional expert-based evaluation frameworks. The paper poses the following research questions: how does “best practice” (e)valuation in online communities differ depending on whether they are produced by community members or experts? And what role play these two practices of valuation for online community performance?

The paper is based on a three-year ethnographic study of a large-scale online community initiative run by the European Commission. Participant observation of online and offline activities (23 events) was complemented with 73 semi-structured interviews with 58 interviewees. The paper draws on Science and Technology Studies, and in particular actor-network theory.

Promoting the idea of “best practice” is not just an exercise about determining what “best” is but rather supposes that best is something that can travel across sites and be replicated. The paper argues that it is crucial to understand the work performed to coordinate multiple practices of producing “best practice” as apparatuses of valuation. Hence if practices are shared or circulate within an online community, this is possible because of material-discursive practices of dissociation and association, through agential cuts. These cuts demarcate what is important – and foregrounded – and what is backgrounded. In so doing new “practice objects” are produced.

The research was conducted in the European public sector where participants are not associated through shared organisational membership (e.g. as employees of the same organisation). An environment for determining “best practice” that is limited to an organisation’s employees and more homogeneous may reveal further dynamics for “best practice” production.

This paper sheds light on why it is so difficult to reach commensuration in crowd-sourced environments.

The paper provides an analysis of how online community members collaborate in order to identify relevant and meaningful user-generated content. It argues that “best practice” is produced through a process of commensuration.