Motivation and performance of user-contributors: Evidence from a CQA forum

Highlights

• We track the activities of a cohort of new Yahoo Answers users over 12 months.

• Predictions from contest theory are tested against these users’ behavior on Yahoo Answers.

• For users who are more contest driven, flexible hours are associated with higher performance.

• But for these same users, multitasking across subjects is associated with lower performance.

Abstract

There is an increasing number of Collaborative Question Answering (CQA) websites and a growing reliance of the online community on user-generated content. In this paper, we study users’ motivation to win social recognition contests (best answers) and how multitasking and flexible hours influence the rate of winning contests. Heterogeneity of contest motivation is estimated at the user level in a standard contest framework and used to demonstrate that, for those who are more highly motivated to win contests, multitasking across subjects is associated with a lower performance rate, while flexible hours are associated with a higher performance rate.

Introduction

Online forums have become an important part of the crowdsourcing phenomenon. In particular, public Collaborative Question Answering (CQA) forums rely on user-contributed questions and answers, which attract visitor traffic that enables the platform to generate potential revenues and to benefit society by quickly providing answers to millions of people.¹ One of the limitations of the literature that investigates crowdsourcing phenomena, or user behavior on the Internet more generally, has been the absence of a microeconomic theory of user behavior. The goal of this paper is to apply known economic theory to understand user behavior in CQA forums.

Specifically, we focus on the following questions: (a) What is the appropriate measure of a CQA forum user’s motivation? (b) What is the relationship between this measure and the user’s performance in a CQA forum? A typical feature of online CQA forums is some type of social recognition system. For instance, in the forum we study (Yahoo! Answers), users earn two “points” by posting an answer to a question and ten “points” if that answer is selected as the Best Answer (BA), either by the person who originally posted the question or by a community vote, if left unchosen by the original poster. Points are cumulative, and users attain higher “levels” (from 1 to 7) based on their total accumulated points. A user’s point total signals their social status on the forum.²

Such social recognition systems effectively create contests among users (i.e., users compete to win a BA), and our basic idea is to measure the user’s contest-driven motivation (as opposed to other sources of motivation) based on a theoretical relationship between the sizes of their own and others’ contributions in equilibrium.³ For two reasons, we expect contest motivation to play an important role in this forum. First, our conversations with insiders reveal that Yahoo! Answers has evolved into more of a point-gathering game than a marketplace where knowledge exchange is priced by using the points as currency to ask questions. That is, a large fraction of users seems to be motivated to win BAs in order to accumulate points faster and attain higher levels, as well as higher BA win rates.

Second, from the platform’s standpoint, the quality of posted answers matters to sustain user viewership, and it is generally thought that CQA forums employ a contest-like feature to encourage users to post high-quality answers (e.g., Morgan and Wang, 2010). Specifically, in Yahoo! Answers the BA award is given to only one answer per question. As shown by Lazear and Rosen (1981), for instance, this type of incentive structure can provide optimal incentives, where the key to motivation hinges on the difference between the highest and second-highest rewards. The eight-point difference between those who win a BA (10 points) and those who do not (2 points) can be expected to provide meaningful incentives only if users care about winning those points, that is, only if the users are contest-motivated.

Uncovering the effects of the social recognition system, however, can be challenging at the aggregate (e.g., question thread) level. This is because the contest system did not change over time in a significant way, which leads to little variation in the amount of answers provided at the question level. Instead, contest motivation should properly be measured at the individual user level, because users are often differentially motivated by the social recognition system and, hence, respond to the contest reward to varying degrees. Thus, we focus on the effects of the contest system on user performance by tracking individual users’ answering activities on the forum, rather than the stock of total answers provided at the question level.

We begin the analysis by measuring the degree of contest motivation by estimating separate regressions, one for each user, where the observations in each regression are the answers the user has posted on the forum. Henceforth, we refer to these regressions as the “motivation” regressions. We estimate these regressions individually for each user because we want to uncover the sensitivity of each user’s answer contributions to the total stock of posted contributions. This slope can, in theory, differ across users. And we find that, empirically, it does indeed, which highlights the importance of estimating it separately for each user. A strength of the data is that there are sufficient observations for each individual (i.e., different answering activities) to estimate a separate motivation regression for each user.

After measuring the degree of contest-motivation for each user in the motivation regressions, we estimate a single, cross-sectional regression to examine the relationship between the aforementioned measure of contest-motivation and the user’s BA hit rate, which is defined as the number of BAs divided by the number of answers provided by the user. The observations in that regression, to which we refer as the “performance” regression, are thus the individual users.

In the context of the performance regression, it would be comforting for the platform designers to discover that, indeed, users who are more contest driven tend to score higher BA hit rates, because BAs are presumably of higher quality than non-BA winning answers. However, CQA platforms can have other features that might significantly affect user behavior. Thus, we must understand how the user’s contest motivation interacts with other salient determinants of user performance. We address this issue by including, on the right-hand side of the performance regression, some behavioral factors as controls and interactions with the motivation parameter derived from our motivation regressions.

The behavioral factors on which we focus are “multitasking” (i.e., answering in multiple subject categories rather than specializing in one or few) and “flexible hours” (i.e., answering on multiple days or hours, as opposed to concentrating answers on particular days or hours). We borrow these concepts from the modern “job design” literature, which highlights them to provide contrast with traditional jobs that require labor specialization and rigid hours.

Our finding is that our measure of contest motivation is positively associated with performance. Additionally, measures of multitasking are negatively correlated with performance, whereas measures of flexible hours are positively correlated. We also find that the more highly-motivated users tend to exhibit lower performance when they multitask, while they tend to exhibit higher performance under flexible hours.

This study relates to a few streams of literature. As previously mentioned, the premise that the point-award system can motivate at least some of the users to provide high-quality answers is based on classic contest (or tournament) theory. A number of experimental studies have shown that rank-order tournaments are, in fact, more effective for stimulating individual performance compared to other pay systems like piece rates, where absolute rather than relative performance matters (e.g., Bull, Schotter, Weigelt, 1987, van Dijk, Sonnemans, van Winden, 2001). However, in those studies the variance of effort (across players) in tournaments is often quite large compared to what would occur under other incentive schemes. This suggests that there may be significant unobserved heterogeneity among contestants; the level of motivation is one such source of heterogeneity that we aim to investigate.

More recently, the optimal design of contests has received growing theoretical and practical interest from both researchers and practitioners.⁴ However, whether contest theories can be applied to online platforms’ virtual reward systems is not well understood. For instance, Gallus (2016) shows that symbolic awards randomly assigned to new contributors to the German-language Wikipedia have a considerable effect on user retention. This is a large-scale contest wherein some 150 out of 4000 new contributors were chosen as winners of the award, hence, the environment differs from the large number of small-scale contests in Yahoo! Answers. Further, we focus on the user’s performance, as measured by the BA hit rate, rather than on user retention. Both concepts are of interest to the platform operators, so we see these studies as complementary.

There is also a broader literature on CQA forums, not particularly relating to reward systems. This literature examines the determinants of whether or not a user answers a series of questions depending on the proximity of the user to either the question or the person who posted the question (e.g., Haas, Criscuolo, George, 2015, Hwang, Singh, Argote, 2015). Such studies require data on a complete set of dyads, which typically come from a proprietary firm’s employee knowledge-sharing forums. We think that Yahoo! Answers is somewhat different in that the forum is publicly accessible, so the user does not know a meaningful fraction of other users who may post answers and/or questions. Further, we do not have data on which questions a user chose not to answer. Thus, our analysis can only address issues that condition on a user actually posting an answer to a question.

Our work also relates to the literature on open-source software or other expert forums. For instance, Lerner and Tirole (2002), Lakhani and von Hippel (2003), and Roberts et al. (2006) survey software developers to elicit their levels of motivation in contributing to open-source software projects. This literature often finds user motivations that relate to labor-market related outcomes. For instance, Xu et al. (2016) provide a quantitative analysis of another CQA forum, Stack Overflow, where the researchers examine the changes in contribution activities before and after the user’s actual job changes; they find evidence consistent with career-related motivation. The differences between Yahoo! Answers and open-source software or other expert forums is that the former is unlikely to lead to a career change.

Finally, we contribute to the burgeoning literature on user-generated content. That literature spans multiple disciplines, and we do not attempt to survey it here. One of the distinguishing features of our study compared to that literature is that we focus on user heterogeneity rather than on subject heterogeneity. That is, prior work examines user-contributed data (e.g., review ratings) centered on individual products (e.g., Chevalier, Mayzlin, 2006, Godes, Silva, 2012, Moe, Schweidel, 2012) because there are only a few observations per contributor. Since our interest lies in the user’s motivation and performance in a CQA forum rather than on the effects of reviews on sales, we track a set of user cohorts to get a sufficient volume of data in order to account for the aforementioned user-level heterogeneity by estimating the utility weight a user places on the contest reward.⁵