1 Introduction

Crowdsourcing is now an established research topic and domain of practice. By exploiting Web-mediated communication (e.g. social networks) and labour (e.g. Amazon Mechanical Turk) platforms, requesters engage with individuals and communities in order to find contributors willing to execute a given activity.

Knowledge Crowdsourcing (KC) is a type of crowdsourcing where the tasks to be executed require contributors to possess knowledge or mastery in a given domain of knowledge, in order to successfully contribute. Artwork annotation is a known example of KC task, as it demands for contributors to understand the abstract, symbolic, or allegorical interpretation of the reality depicted in the artwork, and to identify and recognise the occurrences of visual classes (e.g. plants, animals, objects) in the artwork. We refer to these individuals as expert contributors (or experts), to highlight their familiarity with the targeted domain of knowledge.

Online marketplaces provide continuous access to large amount of contributors that are engaged by monetary rewards, and therefore willing to quickly perform the proposed activities. As the suitability of these contributors to the task at hand is typically unknown in advance, recent research proposed several strategies, e.g. worker self-selection and preliminary assessments, to identify expert contributors [4, 11, 12] within a marketplace. While being effective, these solutions showed an intrinsic limitation of using paid crowdsourcing for KC tasks: the variety of expertise available in online marketplaces is limited by the socio-economical composition of their workforce, which inevitably limits the amount of expert contributions that could actually be identifiedFootnote 1. In this context, online social platforms emerge as a viable alternative source of contributors [1]. They (i) enable the interaction with large amount of individuals – potentially orders of magnitude larger than the ones available in online marketplace; and, given their general purpose, (ii) they are more likely to host expert contributors. Previous work focused on the identification of expert contributors for KC tasks, building on approaches that exploited social ties [2], topic-based profiling [6], contextual properties (e.g. geographical location) [5], or Web content consumption [9]. How to invite expert contributors to KC tasks? How to engage them with appropriate rewards? How to create engaging and viral KC campaigns in a replicable manner is still an open research question.

Original Contribution. We advocate the need for a better understanding of how expert contributors could be invited to participate in KC campaigns, as a first step towards their long-term engagement with the requesters’ goals. To this end, we contribute an experiment in expert contributors invitation focused on KC tasks. We seek answer to the following research question:

How can expert contributors drawn from online social platforms be successfully invited to participate in knowledge crowdsourcing tasks?

We depart from previous work by focusing on the crowd invitation problem. Our ultimate goal is to distill robust invitation strategies to be used by requesters in order to tap the latent workforce readily available in open communities. We focus on reddit, a popular social bookmarking platform where users organise in communities to engage in discussions about a broad spectrum of knowledge domains. There, we seek expert contributors in two distinct domain of knowledge – namely botany and ornithology, to contribute an artwork annotation KC task. We study the performance of two invitation strategies: one addressed to the individual expert contributors, and one addressed to communities of knowledge. We measure their effectiveness in terms of (i) engagement with the requester, (ii) interest in the proposed task, and (iii) engagement with the task. Our findings show that direct invitation messages can result in more interest from expert contributors, while community invitations yields, in the context of our experiment, greater amount of contributions.

The remainder of the paper is organised as follows. Section 2 describes our experimental methodology; Sect. 3 reports the result of the study, and discusses our findings; and Sect. 4 concludes.

2 Experimental Methodology

The experiment is organised in three steps. First, we identify communities and expert contributors in reddit that are knowledgeable in the two targeted domains of interest, namely botany and ornithology. This process is described in Sect. 2.1. Then, we dispatch messages of invitation to a knowledge crowdsourcing campaign. We study the performance of two strategies (described in Sect. 2.2), one directly addressed to individual expert contributors, and one collectively addressed to members of relevant communities. The content annotation platform set up for the experiment is presented in Sect. 2.2, while the performance evaluation metrics are introduced in Sect. 2.3.

2.1 Identification of Expert Contributors and Communities

Users in reddit contribute by creating their own submissions, or by commenting and voting existing submissions or comments. reddit is organised in more than 853 K collections called subreddits, each themed to a specific topic, e.g. /r/flowers. Moderators (community voted administrators) keep collections on-topic, according to both general and collection specific rules. Submissions are described by a title and a textual content. A message is directed to a reddit user, and it allows a message (formatted in Markdown, no images) with up to 10 K characters.

We capitalised on the reddit datasetFootnote 2 described in [10], which includes 1.367.276 resources from 491.572 active users. With the aim of including only qualified candidates, we first filtered the original set of resources by preserving the ones (i) featuring at least 20 distinct words and 5 sentences; and (ii) having a domain matching score [10] greater than 0.2 for the two investigated domains. This resulted in 170 K resources, produced by 38 K users in 6 K subreddits.

To identify communities relevant to the targeted knowledge domains, we assigned a score to each subreddit by calculating the cumulative sum of domain matching scores of their resources; we then considered the top 50 subreddits in the resulting ranks, granted that they contained at least 10 contributions from more than one user.

The final pool of candidate experts was composed as follows. We downloaded the full set of resourcesFootnote 3 created by each of the 38 K users. Then, using the same definition of affinity score as in [10], we calculated the score of each resource, and assigned to each user the highest score amongst the ones in her set of resources. Finally, preserved users that: (i) had at least one submission in one of the top 50 relevant subreddits identified in the previous step; and (ii) possess a score higher than 0.2. The process produced in 1301 expert contributors in the botany domain, and 1111 expert contributors in the ornithology domain.

2.2 Invitation Strategies

An invitation strategy operationalises in the dispatch of a message, inviting an expert contributors to perform a domain-specific knowledge crowdsourcing task, namely the annotation of flowers and birds in artworks.

The invitation messages sent to individuals and to communities are depicted in Fig. 1. The message contains a description of the sender (the first author of the paper), the ongoing research project, the required task, and time constraints. The message also contains a link to a page describing the project in more details, and a link to the content annotation platform.

Each expert contributor received a direct invitation message, personalised on the targeted domain of knowledge, in her reddit inbox. Community invitation messages have been published as a contribution in selected subreddits, for all the community members to see and react upon (i.e. via voting and commenting).

Fig. 1.
figure 1

Invitation messages. Best viewed in the electronic version

Full size image

We performed a study on the CrowdFlower human computation platform, aimed at validating the quality of the invitation messages w.r.t. its intended purpose. Crowd workers were asked to imagine themselves as knowledgeable in the targeted domain of knowledge, and to express their evaluation about the message on a 5-point Likert scale. The evaluation included the following dimensions: Friendliness; Clarity; tone (colloquial or formal; perceived emotion of the invitation (impersonal, personal); likeliness to respond (How likely is it that you, as someone [...], will react to the invitation?) likeliness to contribute (How likely is it that you, as someone [...], will start the task and contribute you knowledge, based on this invitation?).

The task was addressed to high quality USA workers (Level 3), to acquire truthful responses from speakers of the same language as our target platform. We payed $0.05 per evaluated message, and set up 120 executions. On average, workers perceived the message as friendly (\(\mu =4.4\), \(\sigma =0.73\)) and clear (\(\mu =4.6\), \(\sigma =0.60\)), moderately personal (\(\mu =3.5\), \(\sigma =0.99\)), but with a slightly formal tone (\(\mu =3.7\), \(\sigma =0.92\)). Given the invitation message, workers would be inclined to respond (\(\mu =3.9\), \(\sigma =1.16\)) and contribute (\(\mu =3.9\), \(\sigma =1.13\)). Workers left useful feedback – (i) explain the benefit for contributors; (ii) to explain the required amount of time; (iii) expand the project description – that we incorporated in the final version of the message.

Content Annotation Platform: Accurator. We built upon the Accurator content annotation framework [7]. We included an introduction page, containing a detailed explanation of the goals of our research project (including a video), a short description of the task, and a button to start the task (Fig. 2a). The page was used to record statistics about users following up on the invitation. The Accurator annotation page contained instructions (Fig. 2b), optional field for users to self-report their username and knowledge level, and artworks to be annotated (Fig. 2c). Each direct invitation message contained a personalised link, unique for each user, to the introduction page. Both the invitation message and the information page contained a personalised link to the annotation page. The community invitation messages contained anonymous links. For both strategies, a user could perform any number of artwork annotations, and stop and continue non-completed tasks any time, using the provided link, until the deadline (two weeks) mentioned in the invitation.

Fig. 2.
figure 2

Content annotation platform. Best viewed in the electronic version

Full size image

2.3 Evaluation Metrics

To study the performance of two invitation strategies, we measure their effectiveness using three classes of metrics. In the case of communities, we assume the amount of potential users to be the number of subscribers to a given subreddit. This is indeed an overestimation, as not all users subscribed to a subreddit are also active users.

First, we measure the engagement of expert contributors with the requester in terms of number of replies from invited users (#Res). We use this metric to quantify if and when users feel compelled to interact with the requester, to engage in discussions about the invitation itself, or about the task at hand. We also provide qualitative observations about the obtained replies. Second, we address the interest of candidate experts in the proposed task. We consider the invitation read conversion rate (IRCR), i.e. the ratio of invited users who read the invitation, to measure the effectiveness of the strategy in terms of stimulated “curiosity” in proposed activity; the invitation execution conversion rate (IECR), i.e. the ratio of users who opened the execution page, to measure the effectiveness of the strategy in attracting potential contributors; and the invitation to reading response time (IRRT), i.e. the average time between sending the invitation and reading the introduction or the execution page. Finally, the third class of measures addresses the engagement of expert contributors with the task. We consider the invitation to contribution ratio (ICR), i.e. the ratio of users who performed at least one annotation; and the contribution size (CS), i.e. the number of completed tasks.

3 Analysis

The experiment took place in two distinct time phases. The first phase addressed the first invitation strategy, i.e. the dispatch of individual, personalised invitation message, and took place between August 22nd and September 26th 2015.

To account for the limitations of Reddit APIFootnote 4, messages were sequentially delivered, in decreasing order of candidate expert’s matching score, and with a random delay between each other. The second invitation strategy, targeting communities, was experimented on January 31st 2016, and we gathered log data for a period of two weeks. The long delay between the two experimental phases was planned to minimise learning effects within the selected population. We selected 5 subreddits from each domain, picked from the list of top 50 candidate domains, and reported in Table 1. We found no user overlap between the invited usernames in the first strategy and the self-reported usernames in the second strategy. Table 2 summarises obtained results. We report distinct figures for each considered domain and invitation strategy.

Table 1. Selected subreddits for community invitation, rank, and # of subscribed users.
Full size table
Table 2. Experimental results. Metrics are described in Sect. 2.3.
Full size table

Engagement with the Requester. The invitation messages triggered diverse responses, both according to the domain of knowledge and invitation strategy. Individuals contacted about a flower-related tasks replied the most, while birds-related community members expressed more interest in our contribution to their subreddits. To better explain such differences, we manually classified each reply into one or more categories, including: (issues with) the selection process; questions about the project; questions about the annotation task; and intentions to contribute. All replies were friendly and constructive, indicating a good attitude towards our initiative. No reply asked for additional information about the project or about the annotation task. \(15\,\%\) of users in the flower domain and \(0.07\,\%\) in the birds domain acknowledged the reception of the message, and promised to inspect the task; interestingly, all visited the introduction page, and none visited the annotation page.

The majority of replies related to the selection process (\(70\,\%\) in the flower domains, \(57\,\%\) in the birds domain), where the major concern was about the wrong attribution of expert capabilities. An inspection of the matching domain scores for these users shows their belonging to the whole spectrum of the users rank. This result provides two interesting observations: (i) users felt compelled about being misclassified as experts; and (ii) a mistake from our side was sufficient to establish a communication channel, despite the unsolicited nature of our message. The remaining replies consisted of responses that were not related to the project, and did not hint to an opinion/interest from the user (e.g. a polite “no thank you”).

Comments and replies to our community invitation messages were less frequent, but mostly focusing on more details about the project and the annotation task. The limited number of responses does not allow for relevant observations. However, as we will see in the next section, the lack of replies did not translated into a lack of interest in the task: simply put, users were not compelled to interact with us on reddit, but were triggered by our call for contribution.

Interest in the Proposed Task. The conversion rates of our strategies are promising both for invitation read and invitation execution: \(10\,\%\) of experts contributors from the flower domain accessed the introduction page, the annotation page, or both; the number drops to \(3\,\%\) in the birds domain, although the number of identified expert contributors was \(15\,\%\) lowers. Community invitations were also successful, despite the lower percentages in Table 2. Respectively, 100 and 119 users from the targeted flower and birds communities visited the introduction page, and 93 and 110 respectively visited the annotation page. Given the unique nature of our study, it is not possible to compare our results to previous workFootnote 5. Finally a note about the invitation to reading response time. While we observe great variability, average values are at least in the order of hours; this result suggests that this sort of expert contributors invitations is not suitable for applications requiring low latency and quick response times.

Engagement with the Task. Invitation conversion rates were relatively low. No more than 20 users – in each configuration of strategy/domain – actually performed at least one annotation task. These numbers, however, yield a \(10-20\,\%\) conversion rate from users entering the execution page to users actually annotating an artwork. This result is promising, when compared to the one reported in [9], where authors adopted target advertising for recruiting, and monetary incentives to keep contributors engaged. Community invitation strategies provided a considerably higher amount of complete annotations, hinting toward the ability to attract more productive users. We hypothesis this difference to be mainly due to the characteristics of the reddit platform, which facilitates community behaviour in favour of direct communications between members. The validation of this hypothesis is left to future work.

Threats to Validity. The target platform (reddit), due to unknown spam detection mechanisms, could have removed direct messages we have send. Only when users responded, i.e. clicked a link or replied, we are sure this did not occur. The chosen dataset and identification model used in the first strategy affects the outcome of this strategy. Using a larger dataset and optimizing the model will most likely increase the response rate of the first strategy but will have a no effect on the second strategy. Researchers repeating this experiment should take this into account.

4 Conclusion and Future Work

This paper contributes an experiment aimed at assessing two invitation strategies for expert contributors. We discuss the performance of a strategy directly addressing individual contributors, and of a strategy addressing communities of knowledge. We provide several novel insights. For instance, we observed how an individual invitation strategy yields more interest to a knowledge crowdsourcing task, while higher amount of task executions were obtained from community invitations.

This work shows how future work can develop in multiple directions. Our ultimate goal is to distill robust invitation strategies to be used by requesters in order to tap the latent workforce readily available in open communities. In this respect, we will investigate how to account for the expert contributor’s level of knowledge in the creation of personalised messages. We will also investigate the performance of similar strategies in other online platform (e.g. Quora, StackExchange), and conduct experiments involving domains of knowledge of diverse diffusion in the general population.