Skip to main content
Springer Nature Link
Log in

Some Pitfalls to Beware When Applying Models to Issues of Policy Relevance

  • Chapter
  • First Online:
Simulating Social Complexity

Part of the book series: Understanding Complex Systems ((UCS))

Abstract

This chapter looks at some of the ways things can go wrong when mathematical or computational models are applied to inform policy on important issues. It looks at some of the pitfalls in the model construction and development phase, including choosing assumptions, the effect of ‘theoretical spectacles’, oversimplified models, not understanding model limitations, and not testing a model enough. It then goes on to discuss the pitfalls that can occur when a model is applied to inform policy, including entrenched policies based on models with little or no evidential support and how models can narrow the evidential base considered. It also looks at confusions concerning model purpose and kinds of question they may answer, when models are used out of context, asking unreasonable things of models, when the uncertainties are too great, when models give a false sense of security, and when the focus should be on values rather than facts. This discussion is then illustrated with two examples, one economic and one from fisheries. It concludes that most of these problems stem from the interface between the modelling and policy worlds. It ends with some simple recommendations to reduce these mistakes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    At best, examination is by a few in the same domain as themselves—people who likely have the same assumptions and worldview. Thus, many models are not effectively critiqued in an independent manner.

  2. 2.

    This is shorthand for saying the model’s assumptions make a model useless in terms of its purpose.

  3. 3.

    http://www.berkshirehathaway.com/letters/2008ltr.pdf (accessed 1 June 2017)

  4. 4.

    Giampietro and Saltelli (2014) provide a discussion on these questions in relation to the ecological footprint.

  5. 5.

    However, model-based explanations of why climate change has been happening are well founded.

  6. 6.

    Whilst models are a tool rather than a picture (see Chap. 4), some are so useless at what they are supposed to do, that it makes sense to call them wrong.

References

  • Bavington, D. (2009). Managing to endanger: Creating manageable cod fisheries in Newfoundland & Labrador, Canada. Maritime Studies (MAST), 7, 99–119.

    Google Scholar 

  • Bavington, D. (2010). Managed annihilation: An unnatural history of the Newfoundland cod collapse. UBC press.

    Google Scholar 

  • Bavington, D. (2015). Marine and freshwater fisheries in Canada: Uncertainties, conflicts, and hope on the water. In B. Mitchell (Ed.), Resource and environmental management in Canada. Canada: OUP.

    Google Scholar 

  • Benessia, A., Funtowicz, S., Giampietro, M., Guimarães Pereira, A., Ravetz, J., Saltelli, A., et al. (2016). Science on the Verge. Tempe, AZ: The Consortium for Science, Policy and Outcomes at Arizona State University.

    Google Scholar 

  • Black, F., & Scholes, M. (1973). The pricing of options and corporate liabilities. Journal of Political Economy, 81(3), 637–654.

    Article  MathSciNet  MATH  Google Scholar 

  • Bruff, I. (2016). Scandalous obfuscations of crisis in Europe. In: Progress in political economy (PPE). Available from: http://ppesydney.net/scandalous-obfuscations-of-crisis-in-europe/.

  • Cavero, T., & Poinasamy, K. (2013). A cautionary tale: The true cost of austerity and inequality in Europe. Oxford: Oxfam International.

    Google Scholar 

  • Costello, C., Ovando, D., Clavelle, T., Strauss, C. K., Hilborn, R., Melnychuk, M. C., et al. (2016). Global fishery prospects under contrasting management regimes. Proceedings of the National Academy of Sciences, 113(18), 5125–5129.

    Article  Google Scholar 

  • Edmonds, B. (2017). Five different modelling purposes. doi:https://doi.org/10.1007/978-3-319-66948-9_4.

    Chapter  Google Scholar 

  • Finlayson, A. C. (1994). Fishing for truth: A sociological analysis of northern cod stock assessments from 1977 to 1990. St. John’s, Newfoundland: Institute of social and economic research, Memorial University of Newfoundland.

    Google Scholar 

  • Finley, C. (2008). A political history of maximum sustainable yield, 1945–1955. In D. Starkey, P. Holm, & M. Barnard (Eds.), Ocean's past: Management insights from the history of marine animal populations (pp. 1989–1206). London: Earthscan.

    Google Scholar 

  • Giampietro, M., & Saltelli, A. (2014). Footprints to nowhere. Ecological Indicators, 46, 610–621.

    Article  Google Scholar 

  • Glynn, P. D. (2015). Integrated environmental modelling: Human decisions, human challenges. Geological Society, London, Special Publications, 408(1), 161–182.

    Article  Google Scholar 

  • Harris, L. (1990). Independent review of the northern cod stock: Executive summary, and recommendations, Available at: http://www.dfo-mpo.gc.ca/Library/114277.pdf.

  • Herndon, T., Ash, M., & Pollin, R. (2013). Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff. Cambridge Journal of Economics, 38(2), 257–279.

    Article  Google Scholar 

  • Holbraad, M., Pedersen, M. A., & de Castro, E. V. (2014). The politics of ontology: Anthropological positions. Cultural Anthropology, 13. Available at: https://culanth.org/fieldsights/462-the-politics-of-ontology-anthropological-positions.

  • Kuhn, T. (1962). The Structure of Scientific Revolutions. Chicago: University of Chicago Press.

    Google Scholar 

  • Moore, J. W. (2017). The Capitalocene, Part I: On the nature and origins of our ecological crisis. The Journal of Peasant Studies, 44(3), 594–630.

    Article  Google Scholar 

  • Pearce, W., Wesselink, A., & Colebatch, H. (2014). Evidence and meaning in policy making. Evidence & Policy, 10(2), 161–165.

    Article  Google Scholar 

  • Rogoff, K., & Reinhart, C. (2010). Growth in a time of debt. American Economic Review, 100(2), 573–578.

    Article  Google Scholar 

  • Rosewell, B. (2017). Complexity science and the art of policy making. In J. Johnson, P. Ormerod, B. Rosewell, A. Nowak, & Y. C. Zhang (Eds.), Non-equilibrium social science and policy (pp. 159–178). Switzerland: Springer.

    Google Scholar 

  • Saltelli, A., & Funtowicz, S. (2014). When all models are wrong. Issues in Science and Technology, 30(2), 79–85.

    Google Scholar 

  • Saltelli, A., Stark, P. B., Becker, W., & Stano, P. (2015). Climate models as economic guides scientific challenge or quixotic quest? Issues in Science and Technology, 31(3), 79–84.

    Google Scholar 

  • Saltelli, A., & Giampietro, M. (2017). What is wrong with evidence based policy, and how can it be improved? Futures, 91, 62–71, Available at: https://doi.org/10.1016/j.futures.2016.11.012.

  • Scott, J. C. (1998). Seeing like a state: How certain schemes to improve the human condition have failed. Yale University Press.

    Google Scholar 

  • Sterman, J. D. (2002). All models are wrong: Reflections on becoming a systems scientist. System Dynamics Review, 18(4), 501–531.

    Article  Google Scholar 

  • Stern, N. (2016). Economics: Current climate models are grossly misleading. Nature, 530(7591), 407–409.

    Article  Google Scholar 

  • Stewart, I. (2012). The mathematical equation that caused the banks to crash. The Observer, 12 February 2012, https://www.theguardian.com/science/2012/feb/12/black-scholes-equation-credit-crunch. Accessed 01 June 2017.

  • Stiglitz, J. E. (2011). Rethinking macroeconomics: What failed, and how to repair it. Journal of the European Economic Association, 9(4), 591–645.

    Article  Google Scholar 

  • Weinberg, A. M. (1972). Science and trans-science. Minerva, 10(2), 209–222.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Policy Modelling, Manchester Metropolitan University, All Saints, Manchester, M15 6BH, UK

    Lia ní Aodha & Bruce Edmonds

Authors
  1. Lia ní Aodha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruce Edmonds
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lia ní Aodha .

Editor information

Editors and Affiliations

  1. Centre for Policy Modelling, Manchester Metropolitan University, Business School, Manchester, United Kingdom

    Bruce Edmonds

  2. Centre for Policy Modelling, Manchester Metropolitan University, Business School, Manchester, United Kingdom

    Ruth Meyer

Further Reading

Further Reading

For further information, we suggest you read the following four reports. Some of these are books, others are articles.

Other useful reading includes the following.

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Aodha, L.n., Edmonds, B. (2017). Some Pitfalls to Beware When Applying Models to Issues of Policy Relevance. In: Edmonds, B., Meyer, R. (eds) Simulating Social Complexity. Understanding Complex Systems. Springer, Cham. https://doi.org/10.1007/978-3-319-66948-9_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-66948-9_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-66947-2

  • Online ISBN: 978-3-319-66948-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics