Abstract
A body of psychological research has examined the correspondence between a judge’s subjective probability of an event’s outcome and the event’s actual outcome. The research generally shows that subjective probabilities are noisy and do not match the “true” probabilities. However, subjective probabilities are still useful for forecasting purposes if they bear some relationship to true probabilities. The purpose of the current research is to exploit relationships between subjective probabilities and outcomes to create improved, model-based probabilities for forecasting. Once the model has been trained in situations where the outcome is known, it can then be used in forecasting situations where the outcome is unknown. These concepts are demonstrated using experimental psychology data, and potential applications are discussed.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Dawes, R.M.: Confidence in intellectual vs. confidence in perceptual judgments. In: Lantermann, E.D., Feger, H. (eds.) Similarity and choice: Papers in honor of Clyde Coombs, pp. 327–345. Han Huber, Bern (1980)
Lichtenstein, S., Fischhoff, B.: Do those who know more also know more about how much they know? The calibration of probability judgments. Organizational Behavior and Human Performance 20, 159–183 (1977)
Wallsten, T.S., Budescu, D.V.: Encoding subjective probabilities: A psychological and psychometric review. Management Science 29, 152–173 (1983)
Cooke, R.M.: Experts in uncertainty: Opinion and subjective probability in science. Oxford University Press, New York (1991)
O’Hagan, A., Buck, C.E., Daneshkhah, A., Eiser, J.R., Garthwaite, P.H., Jenkinson, D.J., Oakley, J.E., Rakow, T.: Uncertain judgements: Eliciting experts’ probabilities. Wiley, Hoboken (2006)
Wallsten, T.S., Budescu, D.V., Erev, I., Diederich, A.: Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making 10, 243–268 (1997)
Yates, J.F., Curley, S.P.: Conditional distribution analyses of probabilistic forecasts. Journal of Forecasting 4, 61–73 (1985)
Keren, G.: On the ability of monitoring non-veridical perceptions and uncertain knowledge: Some calibration studies. Acta Psychologica 67, 95–119 (1988)
Lichtenstein, S., Fischhoff, B., Phillips, L.D.: Calibration of probabilities: The state of the art to 1980. In: Kahneman, D., Slovic, P., Tversky, A. (eds.) Judgment under uncertainty: Heuristics and biases, pp. 306–334. Cambridge University Press, Cambridge (1982)
Price, P.C.: Effects of a relative-frequency elicitation question on likelihood judgment accuracy: The case of external correspondence. Organizational Behavior and Human Decision Processes 76, 277–297 (1998)
Dougherty, M.R.P.: Integration of the ecological and error models of overconfidence using a multiple-trace memory model. Journal of Experimental Psychology: General 130, 579–599 (2001)
Merkle, E.C., Van Zandt, T.: An application of the Poisson race model to confidence calibration. Journal of Experimental Psychology: General 135, 391–408 (2006)
Thomas, A., O’Hara, B., Ligges, U., Sturtz, S.: Making BUGS open. R News 6, 12–17 (2006)
Swets, J.A.: Assessment of NDT systems–Part I: The relationship of true and false detections. Materials Evaluation 41, 1294–1298 (1983)
Swets, J.A.: Assessment of NDT systems–Part II: Indices of performance. Materials Evaluation 41, 1299–1303 (1983)
Hastie, T., Tibshirani, R., Friedman, J.: The elements of statistical learning. Springer, New York (2001)
Batchelder, W.H., Romney, A.K.: Test theory without an answer key. Psychometrika 53, 71–92 (1988)
Erev, I., Wallsten, T.S., Budescu, D.V.: Simultaneous over- and underconfidence: The role of error in judgment processes. Psychological Review 101, 519–527 (1994)
Ratcliff, R., Starns, J.: Modeling confidence and response time in recognition memory. Psychological Review 116, 59–83 (2009)
Gelman, A., Hill, J.: Data analysis using regression and multilevel/hierarchical models. Cambridge, New York (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Merkle, E.C. (2010). Calibrating Subjective Probabilities Using Hierarchical Bayesian Models. In: Chai, SK., Salerno, J.J., Mabry, P.L. (eds) Advances in Social Computing. SBP 2010. Lecture Notes in Computer Science, vol 6007. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-12079-4_4
Download citation
DOI: https://doi.org/10.1007/978-3-642-12079-4_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-12078-7
Online ISBN: 978-3-642-12079-4
eBook Packages: Computer ScienceComputer Science (R0)