Abstract
When can macroscopic data about a system be used to set parameters in a microfoundational simulation? We examine the epistemic viability of tweaking parameter values to generate a better fit between the outcome of a simulation and the available observational data. We restrict our focus to microfoundational simulations—those simulations that attempt to replicate the macrobehavior of a target system by modeling interactions between microentities. We argue that tweaking can be effective but that there are two central risks. First, tweaking risks overfitting the simulation to the data and thus compromising predictive accuracy; and second, it risks compromising the microfoundationality of the simulation. We evaluate standard responses to tweaking and propose strategies to guard against these risks.
Similar content being viewed by others
References
Axelsen B. E., Anker-Nilssen T., Fossum P., Kvamme C., Nøttestad L. (2001) Pretty patterns but a simple strategy: Predator–prey interactions between juvenile herring and Atlantic puffins observed with multibeam sonar. Canadian Journal of Zoology 79: 1586–1596
Barrett, C. L., Beckman, R. J., Berkbigler, K. P., Eubank, S. G., Henson, K. M., Kubicek, D. A., et. al. (2000). TRANSIMS: Transportation analysis simulation system. Los Alamos Unlimited Release (LAUR) 00-1725.
Bearman P. S., Moody J., Stovel K. (2004) Chains of affection: The structure of adolescent romantic and sexual networks. The American Journal of Sociology 110(1): 44–91
Bishop C. (2006) Pattern recognition and machine learning. Springer, New York
Bondi A. (1964) Van der Waals volumes and radii. Journal of Physical Chemistry 68(3): 441–451
Burnham K. P., Anderson D. R. (2002) Model selection and multimodal inference: A practical information-theoretic approach. Springer, New York
Cetin N., Nagel K., Raney B., Voellmy A. (2002) Large-scale multi-agent transportation simulations. Computer Physics Communications 147: 559–564
Dawkins C., Srinivasan T. N., Whalley J. (2001) Calibration. In: Heckman J. J., Leamer E. (eds) Handbook of econometrics. North-Holland, Amsterdam, pp 3653–3701
Eubank S., Guclu H., Anil Kumar V. S., Marathe M. V., Srinivasan A., Toroczkai Z. et al (2004) Modeling disease outbreaks in realistic urban social networks. Nature 429: 180–184
Forster M. R., Sober E. (1994) How to tell when simpler, more unified, or less ad hoc theories will provide more accurate predictions. British Journal for the Philosophy of Science 45: 1–35
Friedman, M. (1953). The methodology of positive economics. In M. Friedman (Ed.), Essays in positive economics (pp. 3–43). Chicago: University of Chicago Press.
Glymour C. (1980) Theory and evidence. Princeton University Press, Princeton
Hausman D. (1992) Why look under the hood?. In: Hausman D. (eds) Essays on philosophy and economic methodology. Cambridge University Press, Cambridge, pp 70–73
Hitchcock C., Sober E. (2004) Prediction versus accommodation and the risk of overfitting. British Journal for the Philosophy of Science 55: 1–34
Hoover K. (2006) A NeoWicksellian in a new classical world: The methodology of Michael Woodford’s interest and prices. Journal of the History of Economic Thought 28(2): 143–149
Kincaid H. (1986) Reduction, explanation, and individualism. Philosophy of Science 53(4): 492–513
Kirman A. (1992) Whom or what does the representative individual represent?. Journal of Economic Perspectives 6(2): 117–136
Kleindorfer G., O’Neill L., Ganeshan R. (1998) Validation in simulation: Various positions in the philosophy of science. Management Science 44(8): 1087–1099
Lucas R. E. (1976) Econometric policy evaluation: A critique. In: Brunner K., Meltzer A. H. (eds) The Phillips curve and labor markets. North-Holland, Amsterdam, pp 19–45
Machamer P., Darden L., Craver C. (2000) Thinking about mechanisms. Philosophy of Science 67(1): 1–25
Mayo D. (2008) How to discount double-counting when it counts: Some clarifications. British Journal for the Philosophy of Science 59: 857–879
Müller P., von Storch H. (2004) Computer modelling in atmospheric and oceanic sciences: Building knowledge. Springer, New York
Nottestad L., Axelsen B. E. (1999) Herring schooling manoeuvers in response to killer whale attack. Canadian Journal of Zoology 77: 1540–1546
Oreskes N., Shrader-Frechette K., Belitz K. (1994) Verification, validation, and confirmation of numerical models in the Earth sciences. Science 263: 641–646
Pierce, S., van Gieson, E. J., & Skalak, T. (2004). Multicellular simulation predicts microvascular patterning. The FASEB Journal, express article 10.1096/fj.03-0933fje. Retrieved December 13, 2010, from http://www.fasebj.org/content/early/2004/03/31/fj.03-0933fje.full.pdf.
Randall D. A., Wielicki B. A. (1997) Measurements, models, and hypotheses in the atmospheric sciences. Bulletin of the American Meteorological Society 78: 399–406
Royall R. (1997) Statistical evidence: A likelihood paradigm. Chapman and Hall/CRC, New York
Rykiel E. (1996) Testing ecological models: The meaning of validation. Ecological Modelling 90: 229–244
Satz D., Ferejohn J. (1994) Rational choice and social theory. Journal of Philosophy 91(2): 71–87
Sokal R. R., Rohlf F. J. (1994) Biometry. W.H. Freeman, New York
van der Waals, J. (1910). The equation of state for gases and liquids (1910 Nobel Prize lecture).
Woodford M. (2006) Comments on the symposium on interest and prices. Journal of the History of Economic Thought 28(2): 187–198
Worrall J. (2002) New evidence for old. In: Gardenfors P. (eds) In the scope of logic, methodology and philosophy of science. Kluwer, Dordrecht, pp 191–209
Author information
Authors and Affiliations
Corresponding author
Additional information
B. Epstein and P. Forber have contributed equally to this work.
Rights and permissions
About this article
Cite this article
Epstein, B., Forber, P. The perils of tweaking: how to use macrodata to set parameters in complex simulation models. Synthese 190, 203–218 (2013). https://doi.org/10.1007/s11229-012-0142-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11229-012-0142-7