Abstract
Linear opinion pools are the most common form of aggregating the probabilistic judgments of multiple experts. Here, the performance of such an aggregation is examined in terms of the calibration and sharpness of the component judgments. The performance is measured through the average quadratic score of the aggregate. Trade-offs between calibration and sharpness are examined and an expression for the optimal weighting of two dependent experts in a linear combination is given. Circumstances where one expert would be disqualified are investigated. Optimal weights for the multiple, dependent experts are found through a concave quadratic program.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bayarri, M. J., & DeGroot, M. H. (1988). Gaining weight: A Bayesian approach. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), Bayesian statistics 3 (pp. 25–44). Oxford: Oxford Univ. Press.
Berger, J. O., & Mortera, J. (1991). Bayesian analysis with limited communication. Journal of Statistical Planning and Inference, 28(1), 1–24.
Brier, G. (1950). Verification of weather forecasts expressed in terms of probabilities. Monthly Weather Review, 76, 1–3.
Byrd, R. H., Nocedal, J., & Waltz, R. A. (2006). KNITRO: An integrated package for nonlinear optimization. In G. di Pillo & M. Roma (Eds.), Large-scale nonlinear optimization (pp. 35–59). Berlin: Springer.
Clemen, R. T., & Winkler, R. L. (2007). Aggregating probability distributions. In W. Edwards, R. Miles, & D. von Winterfeldt (Eds.), Advances in decision analysis. Cambridge, UK: Cambridge University Press.
Clements, M. P., & Harvey, D. I. (2011). Combining probability forecasts. International Journal of Forecasting, 27, 208–223.
Cooke, R. M. (1991). Experts in uncertainty: Opinion and subjective probability in science. Oxford: Oxford University Press.
Degroot, M., & Fienberg, S. (1983). The comparison and evaluation of forecasters. The Statistician, 32, 12–22.
DeGroot, M. H., & Mortera, J. (1991). Optimal linear opinion pools. Management Science, 37, 546–558.
Department of Homeland Security (2006). Bioterrorism Risk Assessment. Biological threat characterization center of the national biodefense analysis and countermeasures center, Fort Detrick, MD.
Destercke, S., Dubois, D., & Chojnacki, E. (2009). Possibilistic information fusion using maximal coherent subsets. IEEE Transactions on Fuzzy Systems, 17, 79–92.
Dubois, D., & Prade, H. (1988). Possibility theory: An approach to computerized processing of uncertainty. New York: Plenum Press.
Faria, A. E. (1996). Graphical Bayesian models in multivariate expert judgments and conditional external Bayesianity. Ph.D. dissertation, Dept. Statistics, University of Warwick.
Faria, A. E., & Smith, J. Q. (1996). Conditional external Bayesianity in decomposable influence diagrams. In J. M. Bernardo, J. O. Berger, A. P. Dawid, & A. F. M. Smith (Eds.), Bayesian statistics 5 (pp. 551–560). Oxford: Clarendon Press.
Faria, A. E., & Smith, J. Q. (1997). Conditionally externally Bayesian pooling operators on chain graphs. Annals of Statistics, 25(4), 1740–1761.
Genest, C., & Zidek, J. V. (1986). Combining probability distributions: A critique and annotated bibliography. Statistical Science, 1, 114–148.
Genest, C., & McConway, K. J. (1990). Allocating the weights in the linear opinion pool. Journal of Forecasting, 9, 53–73.
Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B, 69, 243–268.
Hora, S. C. (2004). Probability judgments for continuous quantities: Linear combinations and calibration. Management Science, 50(5), 597–604.
Hora, S. C. (2011). An analytic method for evaluating the performance of aggregation rules for probability densities. Operations Research, 58(5), 1440–1449.
Hora, S. C., & Iman, R. L. (1989). Expert opinion in risk analysis: The NUREG-1150 experience. Nuclear Science and Engineering, 102(4), 323–331.
Hora, S. C., Hora, J. A., & Dodd, N. (1992). Assessment of probability distributions for continuous random variables: A comparison of the bisection and fixed value methods. Organizational Behavior and Human Decision Processes, 51, 133–155.
Hora, S. C., Fransen, B. R., Hawkins, N., & Susel, I. (2013). Median aggregation of distribution functions. Decision Analysis, 10(4), 279–291.
Kahneman, D., Slovic, P., & Tversky, A. (1982). Judgment under uncertainty: Heuristics and biases. Cambridge: Cambridge University Press.
Matheson, J. E., & Winkler, R. L. (1976). Scoring rules for continuous probability distributions. Management Science, 22(10), 1087–1096.
MATLAB, 7.12, (2011). The MathWorks Inc., Natick, MA.
Morgan, M. G., & Henrion, M. (1990). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis. New York: Cambridge University Press.
National Research Council (2008). The Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change. Washington D.C.: The National Academies Press.
Nesterov, Y., & Nemirovskii, A. (1994). Interior-point polynomial algorithms in convex programming. Philadelphia: SIAM.
Ranjan, R., & Gneiting, T. (2010). Combining probability forecasts. Journal of the Royal Statistical Society: Series B, 72(1), 71–91.
Rasmussen, N., et al. (1975). Reactor safety study: WASH-1400, NUREG-751014. Washington: U.S. Nuclear Regulatory Commission.
Serfling, R. J. (1980). Approximation theorems of mathematical statistics. New York: Wiley.
Shafer, G. (1976). A mathematical theory of evidence. Princeton: Princeton University Press.
Smith, A. F. M., & Makov, U. E. (1978). A quasi-Bayes sequential procedure for mixtures. Journal of the Royal Statistical Society, Series B, 40, 106–112.
Stone, M. (1961). The linear opinion pool. Annals of Mathematical Statistics, 32, 1339–1342.
Walley, P. (1991). Statistical reasoning with imprecise probabilities. UK: Chapman and Hall.
Winkler, R. L. (1969). Scoring rules and the evaluation of probability assessors. Journal of the American Statistical Association, 64(3), 1073–1078.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was supported by the United States Department of Homeland Security through the National Center for Risk and Economic Analysis of Terrorism Events (CREATE) under award number 2010-ST-061-RE0001. However, any opinions, findings, and conclusions or recommendations in this document are those of the authors and do not necessarily reflect views of the United States Department of Homeland Security, or the University of Southern California, or CREATE.
Rights and permissions
About this article
Cite this article
Hora, S.C., Kardeş, E. Calibration, sharpness and the weighting of experts in a linear opinion pool. Ann Oper Res 229, 429–450 (2015). https://doi.org/10.1007/s10479-015-1846-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10479-015-1846-0