Keywords

1 Introduction

Crowdsourcing has spread through many fields in recent years and shown potential for problem solving and innovation creation for different kind of tasks [1]. It has an increasing role in the area of work and knowledge generation for organizations. While only a few years ago students were predominantly crowd workers, the picture is changing and becoming more heterogeneous. In addition, the diversity of crowdsourcing platforms is enormous. This allows the practice of crowdsourcing in many industry sectors and areas. Howe [2] describes the crowdsourcing phenomenon with the outsourcing of tasks “to an undefined generally large group of people in an open call” [2]. Consequently, crowdsourcing delivers opportunities for organizations to widen the range of their organizational boarder and consider other sources to solve problems, accomplish tasks and generate innovation [3]. One of the major benefits for the crowdsourcer is the pool of crowd workers (individual members of the crowd [4]) that can be accessed over crowdsourcing platforms. Furthermore, crowdsourcing platforms enable crowdsourcers to address different types of tasks to crowd workers in open calls. Some tasks can be divided into different parts and distributed to different crowd workers for accomplishment. Other tasks need, due to their complexity, the collaboration of some crowd workers for accomplishment. Collaboration research has shown that the collaboration between different heterogeneous individuals can lead to better outcomes. Especially, if tasks are involved with high complexity level and if they go beyond individuals’ skills to be accomplished [5,6,7,8]. The collaboration between crowd workers is one key to value creation [9]. However, less is known about the collaboration process among crowd workers [9,10,11,12,13] and most research in this field has an explorative focus for a special domain of crowdsourcing. For example, the elaboration on ideas on web platforms to improve the quality of ideas is in the scope of Kipp [14]. Hutter et al. [15] investigate the tension between collaborative and competitive behavior, the balance and the impact of them on the quality of the outcome in design contests. The management of the crowd through the platforms’ process is in the scope of Malhotra and Majchrzak [16] by utilizing the knowledge integration process [17] to guide the crowd workers. However, the collaborative interaction among crowd workers constitutes a gap in prior research [9]. Moreover, the collaboration of crowd workers could lead to better outcomes and help towards preventing or weakening some identified problems on this field. For example, handling enormous user generated input in form of unstructured and/or low quality work with lack of details as well as input that already exists on the platform by recognizing same or similar content and linking them up to each other, finding any weakness in contributions of others and making suggestions for improvement [11]. Therefore, Tavanapour and Bittner [10, 11] introduced the collaboration process design framework for crowdsourcing (CPDF), that considers three different phases to structure the collaboration of crowd workers. In their work they also point out that the CPDF lacks of more detailed process steps to structure the collaboration more accurately. For example according to the CPDF the crowd worker enters the collaboration with other crowd workers in a “prototyping” process step by working on a/n prototype/artefact [11]. Even though the “prototyping” step is evidenced through the research of Tavanapour and Bittner [11], they also point out that more research is needed to explore this deeper. This leaves room for the question of how the participation of crowd workers leads to the process of prototyping an artefact [10, 11]. At this point, more explorative research could shed light to unanswered aspects as shown with the example above. Since the investigation of how far the CPDF is covered on crowdsourcing platforms and covers the collaboration between crowd workers on crowdsourcing platforms was done in recent work (see [11]), we widen our scope to understand the collaboration of crowd workers more in detail and to explore and investigate which process steps the CPDF is still missing. We ask the following research questions for this purpose. Q1: How do crowd workers proceed to submit a solution on crowdsourcing platforms? Q2: Which process steps, functionalities and problems does the CPDF miss with respect to the collaboration among crowd workers on crowdsourcing platforms?

Initially, the CPDF was derived from and is limited to the literature, this is why we use a different source in this research to identify the missing pieces of the CPDF closer to real world crowdsourcing projects with less limitations to answer Q2. Therefore, we need to understand the patterns that lead to a submission on real world crowdsourcing platforms by answering Q1. To answer Q1 and Q2 we conduct a content analysis (according to Mayring [18, 19]) on existing crowdsourcing platforms. For this purpose, we derive a category system [18] and use the categories for the coding process [20] on crowdsourcing platfoms.

This paper is structured as follows: first, we explain the background of the research by mainly zooming to the CPDF more in detail. Second, we present the methodology before third, presenting the findings of the conducted content analysis. Based on the findings, we redesign the collaboration process of crowd workers. Forth we discuss the collaboration process and close with a conclusion.

2 Research Background

The CPDF according to Fig. 1 considers the phases before (Pre Collaboration Phase) and after (Post Collaboration Phase) the actual collaboration process among crowd workers to consider relevant factors that can influence the collaboration [10]. The Pre Collaboration Phase suggests to consider incentive systems to motivate the crowd workers (in S1), deliver guidance in form of instructions to help the crowd workers through the process of contribution (in S2), give opportunities to get access to perquisite knowledge for participation as well as understanding of others’ contribution (S3) and create content for one specific type of participation (S4) [10].

Fig. 1.
figure 1

Collaboration process design framework for crowdsourcing (according to [10, 11])

Full size image

The Collaboration Process Phase (CPP) provides a design of an interaction process among crowd workers with the goal to create a solution collaboratively that satisfies the open call. Starting with S5 – Prototyping – crowd workers elaborate towards creating a first artefact/prototype of a solution, which serves as a basis to get feedback in S6 for improvement, before submitting a final solution. The revise step (S7) either considers the given feedback to improve, finalize and submit the solution (S8) or can be used to go back to S5 and start over with constructing a prototype [10]. At this point, the Post Collaboration Process starts with documenting the process (S9), sharing the special insights that lead to the solution (S10) and learning from the shared experience of others, towards gaining more expertise (S11) [10].

If we zoom into the Collaboration Process Phase, we can see a jump from S4 to S5 (Fig. 1) and with it the entry to another phase and to the collaborative setting. However, this jump leaves room for the question of how the participation of crowd worker leads to the process of prototyping an artefact [10, 11]. Thus, specially the Collaboration Process Phase (Fig. 1) needs more investigation for more accurate matching to real world crowdsourcing needs [11].

3 Design Science Research Approach

This research starts the second iteration of a larger Design Science Research project (DSR) [21,22,23,24] according to the six-steps approach (see Fig. 2) of Peffers et al. [25].

Fig. 2.
figure 2

The six-step DSR approach [11] according to Peffers et al. [25]

Full size image

The “Problem Identification” and “Objectives of a Solution” remain from the first iteration (see [10, 11]). Therefore, the entry point of this research is taking the loopback after the first iteration to the third step “Design and Development” by using the results of previous research to redefine the categories and criteria of a conducted content analysis on running crowdsourcing platforms (see [11]) and continues with the fourth step “Demonstration” by conducting a content analysis on existing crowdsourcing platforms to gain insights on the collaboration among crowd workers. We analyze the results in the “Evaluation” step and propose a redesigned Collaboration Process Phase for the CPDF to answer the research questions (Q1 and Q2). With this paper, we aim to complete the last step “Communication” and close the second iteration. With the redesigned framework, we aim to contribute prescriptive knowledge [23] towards a “theory for design and action” [22].

4 Analysis and Findings

We conduct the content analysis according to Mayring [18] by deriving a category system to code the relevant content on current ongoing crowdsourcing platforms to gather insights on the collaboration among crowd worker towards solutions.

4.1 Content Analysis

Since we posses a category system from a previous study (see [11]), we utilize it in this paper for more accurate process steps based on the previous results. The previous category system considered each step S1-S11 (Fig. 1) of the CPDF as a category to find evidence for each step in ten real world crowdsourcing projects [11]. As the scope of this paper is on the Collaboration Process Phase (CPP) and the identification of missing steps for the collaboration among crowd workers we exclude the Pre Collaboration Phase and the Post Collaboration Phase from this content analysis. This also results from previous research and the observed insights such as lack and potential for missing process steps in the CPP on real world crowdsourcing platforms [11]. Therefore, we still consider the steps of the CPP S5-S8 as categories for the content analysis. We go one step further and consider, based on the data of prior research (in [11]), four new categories (category 1, 3, 5 and 7) additionally to the previous four (category 2, 4, 6 and 8) resulting from S5-S8. The four new categories are mainly constructed to represent the interaction around the steps S5 “Prototyping”, S6 “Feedback” and S7 “Revise” to close the gap in the CPP. The categories are as follows:

  • Category 1 - Before the Prototyping: This category considers the interaction before the creation or elaboration of or on a first artefact/prototype: The criteria for the coding process is to capture all the activities among crowd worker related to an open call before the actual “Prototyping” step S5 in form of posts (all kind of files) and text segments.

  • Category 2 - Prototyping: Activities among crowd workers that lead towards the creation of any kind of first artefacts or prototypes. The criteria is to capture (related) crowd workers contributions of first prototypes and artefacts such as ideas, solutions, designs, source code etc. [11].

  • Category 3 - After the Prototyping before the Feedback: This category considers the interaction after the creation or elaboration of or on a first artefact/prototype and before any feedback by others than the owners is given: The criteria for the coding process is to capture all the activities among crowd workers related to an open call after the actual “Prototyping” step S5 and before the Feedback step S6 in form of posts (all kind of files) and text segments.

  • Category 4 - Feedback: The first created prototype/artefact is in the scope of this category and any provided feedback in any form related to the produced prototype/artefact. The criteria are to cover any kind of positive or negative feedback in form of rating e.g. like or dislike button, voting ranges or scales, in form of text segments or files highlighted improvement suggestions or pointing out weaknesses to any aspects of the prototype/artefact [11].

  • Category 5 - After the Feedback before the Revision: This category captures all the activities before revising the prototype/artefact by considering the feedback. The criteria for coding is to capture all the activities among crowd worker related to an open call after the “Feedback” step S6 and before the actual “Revise” step S7 in form of posts (all kind of files) and text segments.

  • Category 6 - Revise: Activities among crowd workers that lead towards any kind of revised artefacts or prototypes based on the given feedback. The criteria are to capture (related) crowd workers’ contribution of revised (improved) prototypes and artefacts such as ideas, solutions, designs, source code etc. [11].

  • Category 7 - After the Revision before submitting: This category captures all the activities before submitting the constructed solution this far and after considering the feedback to construct a revised solution. The criteria for coding is to capture all the activities among crowd workers related to an open call after the “Revise” step S6 and before the actual “Submit” step S7 in form of posts (all kind of files) and text segments.

  • Category 8 - Submit: “This category captures necessary formalities for submitting a solution. But before that the crowd workers have to meet the needs for submitting” [11]. Relevant are finalized submitted solution to recognize the end of the collaboration among crowd worker with the reached goal to shape a solution that satisfies the open call.

We coded the content of the considered platforms to the categories [20] and considered produced content of crowd workers in form of texts and files [11] and references to them (e.g. comment from crowdsourcer to crowd workers contribution). In the coding process, on each platform, we went through the first ten open calls and gathered the crowd workers’ contributions of any kind and references to them. Additionally, for each of the gathered data, we examined one by one whether they meet the criteria of one category or not.

4.2 The Considered Crowdsourcing Platforms for the Content Analysis

Table 1 shows the platforms we considered for the content analysis. Most of the platforms were selected, because they displayed a certain number of parallel ongoing projects on which the interaction of crowd workers is possible. For our research, the crowding mechanisms as well as the interaction among crowd workers and crowd workers with crowdsourcer are of importance. Some open calls on the considered platforms revealed a high amount of interaction among a multiple number of actors such as crowd workers and crowdsourcer. On most of the platforms, the interaction among actors was restricted to two to five individuals. One reason for this could be the high amount of contributions and the complexity to get an overview of all contributions. We collected the data on each platform from different open calls between February 2017 and March 2018. We conducted the content analyses with data from 13 different platforms. According to Table 1, the diversity of domains is considered with different submitted artefacts.

Table 1. Platforms (P) considered for the content analyses
Full size table

4.3 Findings

Table 2 lists the platforms on which we found content according to the criteria of the categories.

Table 2. Overview corresponding platforms with relevant content for the categories
Full size table

We identified the most insightful contents for categories 1 and 4. For category 7 (“after the revision before submitting”), we hardly found content on platforms except for the ones listed in Table 2. One reason for this might be that on most platforms feedback was available (category 4) but ignored by crowd workers. Therefore, no revision took place and this leads to less content for categories 5, 6 and 7.

Category 3 reveals little room among prototyping and receiving feedback. If a prototype/artefact is constructed, crowd workers tend to wait for feedback.

Category 8 contains all platforms as displayed in Table 2. However, we could not identify any other content than submitted solutions.

At this point, we examine each category in more detail and describe the insights and interaction patterns we found in the conducted content analysis.

Starting with Category 1, the coding of the content belonging to this category reveals different patterns of the first entry point and interaction among crowd workers. In contrast to the “Prototyping” step in the CPDF, crowd workers do not always start with creating an artefact. The results show that either one idea is shared which leads to collaboration (P1, P3, P4, P5, P6, P8, P9, P10, P11, P12, P13) or an ideation process is started in from of a discussion towards an approach (P1, P2, P6, P10, P13). In both cases, the crowd has the opportunity to discuss the ideas and/or rates the ideas by preference. After the discussion and/or rating a small amount of crowd workers, who are interested in further elaboration find a constellation to elaborate on different parts towards a prototype/artefact of a solution (P1, P2, P6, P10, P13).

In some cases (P3, P4, P5, P7, P8, P9, P11, P12), the described process above is not available because the idea is presented directly with a/n prototype/artefact. Before that, we found no interaction or any kind of pattern.

In sum, the entry point for the collaboration process varies from platform to platform and from open call to open call. Different forms of how crowd workers participate lead to a diversity of the entry points at different states of the process (ideation, discussion, ratings, suggestions etc.).

Category 2 considers patterns for the actual “Prototyping”. If no prototype/artefact is presented with the idea, the crowd workers discuss parts of the solution and organize themselves by splitting tasks (P1, P2, P6, P10, P13) and elaborate by contributing the divided tasks towards the prototype/artefact (P1, P2, P3, P5, P6 P7, P9). If all subtasks are accomplished, they merge them to complete the first prototype/artefact (P1, P2, P3, P5, P6, P7, P9).

In Category 3 after the “Prototyping & before the Feedback”, we could identify one specific pattern. Depending on the type of the artefact/prototype, they checked (proof of concept (P2, P5, P6, P7), (re)producing results (P5, P7, P10)), if the prototype/artefact fulfills its purpose (P2, P6) as in an “evaluation”. Based on the result of the evaluation some of the prototypes were adjusted or improved (P2, P5, P7, P10).

Category 4 contains any kind of “Feedback”. The analysis of the coding revealed the existence of three main kinds of feedback givers. First, the crowd itself in form of ratings (e.g. scales, like/dislike), by providing suggestions for improvement, hints of errors or weaknesses in comments on prototypes/artefacts or in discussions about the prototype/artefact (P2, P4, P5, P7, P8, P11, P12). Second, the platform itself (one platform worker) or experienced crowd workers, who provide detailed feedback on the prototype in an early stage of the process (P1, P2, P7, P9, P10, P11, P12). Third, the crowdsourcer, who also mostly delivered feedback, but at a late state of the process (P1, P4, P5, P8, P9, P10, P11, P12, P13).

In Category 5 after the “Feedback & before the Revise”, we could identify patterns regarding the available feedback. The feedback was discussed with the feedback giver and among the crowd workers, who constructed the prototype/artefact (P1, P2, P7, P10, P11, P12). In some cases, the responsible crowd workers for the prototype filtered the feedback for revising (P2, P7).

Category 6 contains any kind of revised prototype/artefact. In this category, the prototypes/artefacts show minor changes. It appears that most of the crowd workers do not invest much effort to shape the prototype with major changes (P1, P2, P5, P7, P10, P11, P12). Even though, valuable feedback is available, it appears that it is not considered for the final submitted solution (P1, P2, P5, P7, P10, P11, P12). Moreover, the feedback from crowdsourcers are more often considered than the feedback from the crowd (P11, P12).

In Category 7 after the “Revision & before the Submission”, the crowd workers evaluated the changes as in category 3, to check whether the prototype still fulfills its purpose (P2, P5, P7). Figure 5 displays the loop back to the “internal evaluation” at this point.

Category 8 contains the submission of a solution. The crowd workers were mostly following the instructions to fulfill the conditions for submission. Finalizing the prototypes/artefacts were mainly in the scope in this category (all platforms P1-P13). No other patterns or content for other patterns were identified.

Figure 6 summarizes the collaboration of crowd workers on the crowdsourcing platforms which we analyzed in real world by merging the patterns we found in the categories as presented in Figs. 3, 4 and 5. The process is initialized by an idea or an ideation process mostly in the form of brainstorming and ends with a submission of a solution to fulfil the open calls’ needs.

Fig. 3.
figure 3

Process pattern in category 1

Full size image
Fig. 4.
figure 4

Process pattern in category 2 “Prototyping”

Full size image
Fig. 5.
figure 5

Process iteration and pattern from category 3, 4, 5, 6 and 7: evaluation, feedback and revise

Full size image

Not in all cases, collaboration among different actors on the crowdsourcing platforms could be observed. According to Fig. 6, there are crowd workers, who tend to have an idea, do not communicate the idea, but elaborate on it and shape an artefact/prototype. These crowd workers either, instead of checking the prototype/artefact or getting feedback, submit the first created prototype/artefact (P2, P3, P6, P9, P10, P13) or they “evaluate” the prototype by themselves, make some changes and then submit it. Thus, no collaboration or interaction is recognized here. In some cases, we detected that feedback is provided, but ignored (P2, P5, P8). Therefore, there might be potential to motivate crowd workers to revise their own work.

Fig. 6.
figure 6

Crowd workers’ real world process to submit a solution on crowdsourcing platforms

Full size image

If we exclude the above mentioned cases, due to the fact that these cases do not show collaboration on crowdsourcing platforms, we can derive a new collaboration process for the CPDF as displayed in Fig. 7. According to Fig. 7 the steps ST1-4 and ST18-20 remain the same as before in the Pre- and Post Collaboration Phase as well as ST16 and ST17 remain the same as before in the CPP (compare to Fig. 1). The new process steps ST5-15, which we derived, are considered for the new CPP. Additionally, we considered the different process entry points of the crowd workers in ST4. Not only ST5 is a possible entry point but also ST6, ST7, ST8 and ST13 as described before. ST13 is the latest entry point to just give feedback to others’ prototype/artefact and work out suggestions for improvements.

Fig. 7.
figure 7

CPDF with new CPP (with renamed (new) steps (ST))

Full size image

Table 3 displays the platforms, that were utilized to derive the content for each of the CPPs’ new process step. Furthermore, it shows which steps are missing on which platforms to improve the collaboration of crowd workers. In sum, content from all 13 platforms could be utilized for the new process steps. According to Table 3 P4 and P6 contributed content to the same steps as well as P11 and P12. In both cases the type of the constructed prototype/artefact is similar. According to Table 1 for P4 and P6 those are ideas and for P11 and P12 designs. At this point, the produced outcome could have an influence on the interaction pattern of crowd workers. This needs more investigation in future research.

Table 3. Overview of the new derived process steps ST5-ST15 of the CPP and the corresponding analyzed platforms (P)
Full size table

Figure 8 presents the number of steps the content from each platform contributed to and for each steps from how many various platforms the content was gained. According to Fig. 8, the majority of platforms (12 out of 13) contains patterns for ST5 and ST6. Content from P2 was considered for 10 out of 11 steps followed by P1 and P9 with 9 out of 11. On the other end, ST7, ST8 and ST11 benefit from content of five various platforms and the content gained from P3, P4 and P8 delivered patterns for 4 new steps. Therefore, those platforms show potential for improvement by considering the missing steps for a more structured and managed collaboration process as presented in the CPP of Fig. 7.

Fig. 8.
figure 8

Comparison of the coded content of platforms with the derived new steps

Full size image

5 Discussion and Conclusion

Based on the content analysis which relies on current crowdsourcing platforms, we could gain insights about real world process of crowd workers from the beginning of an idea generation to submitting a solution regarding an open call. The scope of our research refers to the CPDF and mainly focusses on the collaboration process of crowd workers on crowdcourcing platforms. Therefore, the CPP of the CPDF was considered for the analysis of this research. The process steps of the CPP, at the beginning of this paper, grounded the choice of categories for the qualitative content analysis. With the approach to investigate, if gaps exist between the process steps of the previous CPP, we could explore and understand the collaboration of crowd workers in more detail. Moreover, we could model the real world process of crowd workers creating and submitting a solution (see Fig. 6) to address an open call and in sum answer Q1.

The real world process of crowd workers creating and submitting a solution on crowdsourcing platforms (according to Fig. 6) does not necessarily need collaboration as described before. The crowd workers could skip ST5-ST9 construct a prototype/artefact (in ST10) and submit it (in ST17) or check the solution (in ST11) and than submit (in ST17). If crowdsourcing platforms aim to prevent such process lifecycle to reduce the probability of receiving a high number of early stage or low quality submitted solutions the management of the crowd [26,27,28,29,30,31] and guidance of the crowd [27, 32,33,34,35,36,37,38] towards supporting the process of the new CPP of the CPDF (Fig. 7) is purposeful.

With the gained insights form the categories 1, 2, 3 and 4 as we could identify missing patterns before a first prototype/artefact is presented, how the previous “Prototyping” step of the CPP could be extended to cover the collaboration better and to capture the variety of feedback and feedback givers. The content analysis also revealed that the revision and submission steps of the further CPP remained mostly unchanged. After the elaboration of a prototype/artefact the majority of crowd workers considered the revision of the prototype/artefact with minimum effort, although helpful feedback was provided. At least that is what we can report form this study. More investigation regarding this is needed in future research to get clarity of the reasons why helpful feedback might not be considered by crowd workers with higher effort. Even though, we excluded the Pre Collaboration Phase of the CPDF, considering incentive systems [32, 33, 39,40,41,42,43,44,45,46,47,48,49,50] for crowd workers to revise the prototype/artefact by considering the majority of helpful feedback (e.g. with remuneration for revise) could be one option to address this issue. Incentive systems should not be limited to motivate the crowd workers to just participate in an open call and may trigger their motivation [32, 33, 39,40,41,42,43,44,45,46,47,48,49,50] even at later process state. For example, provoking crowd workers’ motivation with incentive systems for ST15 and ST16 could lead the crowd worker to invest more effort to improve the prototype/artefact.

More investigation is also needed in future research to gain insight on why crowdsourcers’ provided feedback even at a later stage is considered more often than that of the crowds. One reason could be that valuable feedback might be overseen or might not be detected due to an overload of contributions on crowdsourcing platforms. Another reason could be that responsible crowd workers for the prototype/artefact do not have the experience to detect the valuable feedback of the crowd or share a different vision. If crowd workers are willing to provide more effort to improve their prototype/artefact it seems that they are more motivated by crowdsourcers’ feedback, because at the end the crowdsourcer needs to be satisfied with the submitted solution and not the crowd.

With this paper we close the second iteration of the design science research project as described in Sect. 3 (Fig. 2). Our research aims to contribute towards a “theory of design and action” according to Gregor [22] and provides prescriptive knowledge [23] with the real world process of crowd workers creating and submitting a solution (Fig. 6) and a redesigned framework for crowd workers’ collaboration on crowdsourcing platforms (Fig. 7 (which also answers Q2)). The framework can serve to design and deploy collaborative environments and structured collaboration processes for crowd workers on crowdsourcing platforms towards more elaborated and improved submitted solutions.