Improving judgmental adjustment of model-based forecasts

Abstract

Many managers have access to statistical model-based forecasts and can use these to create their own forecasts. To the analyst, who aims to evaluate forecast accuracy, it is usually unknown to what extent managers use those model forecasts. Moreover, in other situations the analyst may additionally not even have access to those model-based forecasts. The present survey reports on recent developments within this context which concern an understanding of the creation of such managers’ forecasts given that model forecasts are potentially incorporated and the evaluation of their accuracy. A variety of further research topics are given.

Introduction

Many managers create their own forecasts of, for example, sales, and usually these forecasts are taken aboard for decision making. There is large literature on how individuals or managers create forecasts, where typically the focus is on judgmental forecasting. An excellent review of the literature is presented in [17]. An earlier study of Fildes and Hastings [20] presents a summary of what is known in this area.

The present paper deals with a specific situation that is usually called “judgmental adjustment”, see a short review of the literature in Section 5.6 of [17]. In this situation there is a statistical model-based forecast and the manager can use this forecast to create her own forecast. Typically, there is an analyst who aims to evaluate the quality of this manager's forecast. There are two important features in this context, and these have been addressed in the recent literature. First, the analyst usually does not know whether the manager has actively considered the model forecast. So, it is known that there is a model forecast, but whether the manager has incorporated it (and how) is unknown to the analyst. Second, it may occur that the analyst does not have access to the model forecast at all. One thing that he can do then is to try to approximate the model forecast using data on potentially relevant variables. The key premise in both situations is that it is important for appreciating the quality of managers’ forecasts to correctly deduce the contribution of the manager from the final forecast. In mathematical terms: when EF_t is the manager's forecast (for, say, sales at t + 1 made at origin t) and MF_t is the associated model forecast, the analyst can hypothesize the relation

$${\mathit{EF}}_t=\alpha +\beta {\mathit{MF}}_t+I_t$$

where α and β are unknown parameters and where I_t is the unpredictable part that may be called “Intuition”. To appreciate the (origins of the) quality of a manager's forecast, the analyst needs to know α and β, and hence I_t. And, when MF_t is not available, the analyst may try to reproduce MF_t by some approximate model.

There is substantial literature that examines the forecast accuracy of managers’ (or experts’) forecasts, given that managers have access to model forecasts. This literature typically focuses on accuracy statistics and on the potential bias in managers’ forecasts. There is some evidence that managers’ forecasts can improve upon pure model-based forecasts [18], but there is also evidence that managers’ forecasts are much less accurate [7], [8], [9], [11].

Further, there are many studies that more closely examine the interaction of models and managers when it comes to forecasting, see [14] for a survey. The spectrum of examination covers the improvement of managers’ forecasts by providing feedback using statistical models; see for example [15], [19]. It also covers the inclusion of managerial information in model-based forecasts; see for example [23], and the judgmental adjustment of model-based forecasts; see for example [1], [24]. Finally, there are studies on the integration of the information that can be combined to give combined forecasts; see for example [21], [22].

A key issue in the interaction of models and managers is that somehow an analyst should know how this interaction works. There are at least three important reasons for that. First, when statistical models are properly specified, they give unbiased forecasts. It is well documented that managers’ forecasts can be biased [7], [8], [9], [11]. This bias is not necessarily bad as forecast accuracy may be improved, but what should be known is the size of the bias and the source of the bias. Second, forecast errors from statistical models can be used to improve the statistical (econometric) model. That is, when systematic patterns in forecast errors exist, one can change the model by including additional variables. When it is not known how managers create their forecasts, there is thus no opportunity to learn from forecast errors. At the same time, when managers’ forecasts do improve on statistical model forecasts, the analyst also cannot learn what causes this improvement and what can make the models better. Indeed, it would be beneficial to learn about the expert's knowledge that the manager has. Third, to evaluate the quality of the managers’ forecasts, one has to rely on statistical tools to see if any forecast gain is systematic or just luck. For this, again, one needs to approximately know what it is that the manager does.

The outline of this paper is as follows. In Section 2, I will present a simple but useful framework to analyze managers’ forecasts, which builds on the above mathematical expression, and which is new to the literature. It is rather stylized, and may differ across various practical situations, but it does help to understand potential managerial behavior. In Section 3, I review the evidence obtained from interviewing managers who actually create their forecasts (when they have access to model forecasts) and the empirical evidence from analyzing actual forecasts. Section 4 deals with the situation where the analyst does not have access to the statistical model forecast and seeks to approximate this with his own model for the particular setting. A detailed and new empirical example concerning airline revenues illustrates the main idea. Section 5 gives a discussion of the items on the future research agenda, and it concludes this paper with a summary of the main findings.