Abstract
Prequential testing of a forecaster is known to be manipulable if the test must pass an informed forecaster for all possible true distributions. Stewart (J Econ Theory 146(5):2029–2041, 2011) provides a non-manipulable prequential likelihood test that only fails an informed forecaster on a small, category I, set of distributions. We present a prequential test based on calibration that also fails the informed forecaster on at most a category I set of true distributions and is non-manipulable. Our construction sheds light on the relationship between likelihood and calibration with respect to the distributions they reject.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Note that any prequential test as defined above is also a prequential test as defined in Shmaya (2008). Shmaya considers tests as functions of sequences of predictions along a realization. A collection of forecast sequences, one for every realization, defines a unique distribution over sequences (Kolmogorov Extension Theorem), however multiple collections may be mapped into the same distribution. These multiple collections differ only in the probabilities assigned conditional on zero probability finite histories. Hence, by failing on an occurrence of such events our test can be defined as a function of forecasts predictions as well.
Specifically, Stewart (2011) requires that the sum of squared differences on the conditionals be bounded.
We are very grateful to Colin Stewart for bringing this to our attention.
References
Al-Najjar N, Weinstein J (2008) Comparative testing of experts. Econometrica 76(3):541–559
Al-Najjar N, Sandroni A, Smorodinsky R, Weinstein J (2010) Testing theories with learnable and predictive representations. J Econ Theory 145(6):2203–2217
Babaioff M, Blumrosen L, Lambert NS, Reingold O (2010) Testing expert forecasts for parameterized sets of distributions. Microsoft research technical report
Dawid A (1982) The well-calibrated Bayesian. J Am Stat Assoc 77(379):605–610
Dekel E, Feinberg Y (2006) Non-Bayesian testing of a stochastic prediction. Rev Econ Stud 73(4):893–906
Feinberg Y, Stewart C (2008) Testing multiple forecasters. Econometrica 76(3):561–582
Foster D, Vohra R (1998) Asymptotic calibration. Biometrika 85(2):379–390
Fudenberg D, Levine D (1999) An easier way to calibrate. Games Econ Behav 29(1–2):131–137
Halmos P (1974) Measure theory. Springer, New York
Kalai E, Lehrer E, Smorodinsky R (1999) Calibrated forecasting and merging. Games Econ Behav 29(1–2):151–169
Lehrer E (2001) Any inspection is manipulable. Econometrica 69(5):1333–1347
Olszewski W, Sandroni A (2008) Manipulability of future-independent tests. Econometrica 76(6):1437–1466
Olszewski W, Sandroni A (2009a) A non-manipulable test. Ann Stat 37(2):1013–1039
Olszewski W, Sandroni A (2009b) Strategic manipulation of empirical tests. Math Oper Res 34(1):57–70
Sandroni A (2003) The reproducible properties of correct forecasts. Int J Game Theory 32(1):151–159
Shmaya E (2008) Many inspections are manipulable. Theor Econ 3(3):367–382
Stewart C (2011) Non-manipulable Bayesian testing. J Econ Theory 146(5):2029–2041
Vovk V, Shafer G (2005) Good randomized sequential probability forecasting is always possible. J R Stat Soc Ser B (Statistical Methodology) 67(5):747–763
Author information
Authors and Affiliations
Corresponding author
Additional information
We are grateful to Colin Stewart for helpful comments and suggestions. Lambert thanks Google Research and the National Science Foundation under grant No. CCF-1101209. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.
Rights and permissions
About this article
Cite this article
Feinberg, Y., Lambert, N.S. Mostly calibrated. Int J Game Theory 44, 153–163 (2015). https://doi.org/10.1007/s00182-014-0423-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00182-014-0423-0