Predicting box office with and without markets: Do internet users know anything?

Highlights

• HSX and Derby game generate reliable estimates of opening weekend box office.

• Participants over predict low-earning films but under predict high-earning films.

• Prediction bias is greater in the Derby (non-market) mechanism.

• Higher budget films, sequels and films featuring stars are more accurately predicted.

Abstract

This study investigates and compares predictions of opening weekend box office revenue from an online prediction game, the Derby, and an online prediction market, the Hollywood Stock Exchange (HSX), using a sample of 141 films released in 2007. Overall, both mechanisms provide accurate predictions of box office outcomes but tend to over-predict small-earning films and under-predict large-earning films. This bias is present across a number of sub-samples disaggregated by film-specific variables. The bias is consistently greater in the Derby game, suggesting that the market mechanism is superior to the non-market mechanism. There is also evidence that larger budget films, sequels and films featuring stars are predicted more accurately in both settings, and that individual-level predictions improve as films spend more time at the box office and as players gain experience.

Introduction

At least since the work of Galton (1907), economists and other social scientists have observed that collectives possess the ability to formulate accurate predictions about unknown future events in a variety of contexts. Sometimes referred to as the ‘wisdom-of-crowds’ or ‘collective-intelligence’ effect, these aggregated predictions have been shown to routinely outperform expert, statistical, and other predictive techniques. Hayek (1945) explicated the role markets may play in this process and, in more recent times, prediction markets have received much attention as powerful tools by which to aggregate widely dispersed information.¹ The aggregation mechanism may take a variety of forms such as a simple survey or poll, through to more exotic mechanisms such as a double auction (e.g. Chen and Plott, 2002) or pari-mutuel design (e.g. Plott et al., 2003, Axelrod et al., 2009).

But while most prediction mechanisms rely on offering real financial incentives, there are many examples of online games and betting markets which attract large numbers of dedicated and enthusiastic participants yet offer no financial incentives – for example fantasy sports betting or virtual stock-markets such as the Hollywood Stock Exchange (HSX). Although some may argue there is little value studying these so-called ‘artificial’ markets (due to the lack of real incentives), others have observed that predictions arising within them are similar to markets where real financial gains are attainable (Pennock et al., 2001, Servan-Schreiber et al., 2004). It would seem that individuals who participate in such markets and games do so for the implicit satisfaction they derive from simply competing against other participants.

Beyond questions pertaining to artificial vs. real markets, there is also an open question as to whether market-based mechanisms actually do provide better forecasts than other mechanisms – for example statistical forecasting, polling or surveying. In the context of elections, Erikson and Wlezien (2008) and Rothschild (2009) show the superiority of polls under certain conditions, and in the context of football, baseball and movies Goel et al. (2010) point out that the benefits of prediction markets are often only marginal and advocate that benefits need to be weighed against costs before considering their implementation over alternative and, often more simple, mechanisms.

This study investigates and compares predictions of opening weekend box office revenue from an online prediction game, the Derby, and an online prediction market, the Hollywood Stock Exchange (HSX) on a sample of 141 films released in 2007. Predicting box office returns is an important task for film producers and investors but has proven difficult in an industry characterised by extreme levels of uncertainty. Such uncertainty manifests in the famous quote of screenwriter William Goldman who declared about the industry: “nobody knows anything” (Goldman, 1983). The causes and consequences of box office uncertainty have been explored by De Vany and Walls, 1996, De Vany and Walls, 1999, De Vany and Walls, 2004 who attribute it to the inherent unpredictability of audience reception and bandwagon effects caused by consumer interaction – both features of markets for information goods such as movies.

While a number of studies have applied statistical techniques for predicting box office outcomes, statistical models are often complicated by data availability and/or reliability, endogeneity of key explanatory variables, and the heavy-tailed nature of box office revenue data.² An alternative approach is to use prediction markets. A number of studies have shown that box office predictions arising from the HSX are accurate even though the market is artificial with no real incentives.³ Pennock et al. (2001) investigate the HSX and show a high degree of correlation between actual and predicted outcomes. Spann and Skiera (2003) similarly conclude that the HSX market has strong predictive power but find that the predictions are biased upwards for small-earning films and biased downwards for large-earning films. Elberse and Eliashberg (2003) use HSX prices as a measure of opening week expected revenue in a dynamic model of demand and supply. Elberse and Anand (2006) study advertising’s impact on HSX prices. Foutz and Jank (2010) find not only strong predictive power of closing HSX prices, but that predictive accuracy is improved when the history of the price is included using functional shape analysis techniques. And, in a more general survey of prediction markets, Wolfers and Zitzewitz (2004) also note the strong correlation of the HSX closing prices and actual revenues.

The primary purpose of this study is to evaluate another, and arguably simpler, prediction mechanism and compare it with the predictions of the HSX market. The Derby game, a discontinued internet-based game formerly hosted on the popular BoxOfficeMojo.com website, involves participants making weekly predictions of weekend box office revenue for a selection of nominated films. Both the Derby and HSX mechanisms share common features in that they are accessible to the entire internet community at zero cost, yet offer no financial incentives to participants. The mechanisms differ in that the HSX is based on a traditional stock market design with an automated market maker whereas the Derby mechanism simply aggregates participant revenue estimates.

This study uses a set of linear regression specifications to compare predicted with actual revenues for the US opening weekend box office revenues on a sample of 141 movies. The predictions are made the day before theatrical release. The value in studying prediction mechanisms in the context of movies derives principally from the potential they offer producers and investors for decision making at the various stages of the production and distribution processes. The very short-range data in this study are not useful for this purpose. However the results can be useful in designing prediction mechanisms to assist in decision making for longer-time horizons.

Consistent with previous studies, results show that the HSX market mechanism provides accurate predictions of opening weekend revenues but with some tendency towards under-prediction of high-earning films and over-prediction of low-earning films. The Derby game also generates reasonably accurate predictions with the same bias. The bias is consistently greater with the Derby game than with the HSX mechanism across a number of sub-sample analyses. There is also some evidence that large budget films, sequels and films featuring stars are more accurately predicted. Both the HSX and Derby mechanisms provide better forecasts than a simple regression of revenues on explanatory film covariates.

The nature of the Derby game data allow investigation of individual and pooled accuracy. The results show that individual participants typically make better predictions for higher budget films and on films playing at more theatres but are less accurate on films with stars. Further, individuals more accurately predict sequels, remakes, reissues and films further into their run. Individuals generally make more accurate predictions as they acquire more experience in the game. In terms of pooled accuracy, and consistent with the results of the comparative exercise, there is some evidence that large budget films are predicted more accurately. Not surprisingly, pooled predictions are also more accurate for films later in their theatrical run.

The remainder of the paper is structured as follows. Section 2 discusses the HSX and Derby prediction data as well as the institutional features of each prediction mechanism. It also presents basic summary statistics of predictive accuracy in relation to various film-specific covariates. Section 3 outlines the empirical methodology and results of three exercises: (i) aggregate-level comparison of the HSX and Derby prediction mechanisms; (ii) disaggregated comparison of the HSX and Derby mechanisms in relation to film-specific covariates; and (iii) analysis of the extended Derby data set of individual and pooled prediction accuracy. Finally Section 4 discusses results and concludes the paper.