Efficient Testing of Forecasts

Abstract

Each day a weather forecaster predicts a probability of each type of weather for the next day. After n days, all the predicted probabilities and the real weather data are sent to a test which decides whether to accept the forecaster as possessing predicting power. Consider tests such that forecasters who know the distribution of nature are passed with high probability. Sandroni shows that any such test can be passed by a forecaster who has no prior knowledge of nature [San03], provided that the duration n is known to the forecaster in advance. On the other hand, Fortnow and Vohra [FV06] show that Sandroni’s result may require forecasters with high computational complexity and is thus of little practical relevance in some cases. We consider forecasters who select a deterministic Turing-machine forecaster according to an arbitrary distribution and then use that machine for all future forecasts. We show that forecasters even more powerful than the above ones are required for Sandroni’s result. Also, we show that Sandroni’s result does not apply when the duration n is not known to the forecaster in advance.