Skip to main content
SpringerLink
Log in

Checking Simulations: Detecting and Avoiding Errors and Artefacts

  • Chapter
  • First Online:
Simulating Social Complexity

Part of the book series: Understanding Complex Systems ((UCS))

Abstract

The aim of this chapter is to simulations. The reader with a set of concepts and a range of suggested activities that will enhance his or her ability to understand agent-based simulations. To do this in a structured way, we review the main concepts of the methodology (e.g. we provide precise definitions for the terms “error” and “artefact”) and establish a general framework that summarises the process of designing, implementing, and using agent-based models. Within this framework we identify the various stages where different types of assumptions are usually made and, consequently, where different types of errors and artefacts may appear. We then propose several activities that can be conducted to detect each type of error and artefact.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Notes

  1. 1.

    By input parameters in this statement, we mean “everything that may affect the output of the model”, e.g. the random seed, the pseudorandom number generator employed, and, potentially, information about the microprocessor and operating system on which the simulation was run, if these could make a difference.

  2. 2.

    The reader can see an interesting comparative analysis between agent-based and equation-based modelling in Parunak et al. (1998).

  3. 3.

    Note that the thematician faces a similar problem when building his non-formal model. There are potentially an infinite number of models for one single target system.

  4. 4.

    Each individual member of this set can be understood as a different model or, alternatively, as a different parameterisation of one single—more general—model that would itself define the whole set.

  5. 5.

    There are some interesting attempts with INGENIAS (Pavón and Gómez-Sanz 2003) to use modelling and visual languages as programming languages rather than merely as design languages (Sansores and Pavón 2005; Sansores et al. 2006). These efforts are aimed at automatically generating several implementations of one single executable model (in various different simulation platforms).

  6. 6.

    See a complete epistemic review of the validation problem in Kleindorfer et al. (1998).

References

  • Axelrod, R. M. (1997a). Advancing the art of simulation in the social sciences. In R. Conte, R. Hegselmann, & P. Terna (Eds.), Simulating social phenomena. (Lecture Notes in Economics and Mathematical Systems, 456) (pp. 21–40). Berlin: Springer.

    Chapter  Google Scholar 

  • Axelrod, R. M. (1997b). The dissemination of culture: A model with local convergence and global polarization. Journal of Conflict Resolution, 41(2), 203–226.

    Article  Google Scholar 

  • Axtell, R. L. (2000). Why agents? On the varied motivations for agent computing in the social sciences. In C. M. Macal & D. Sallach (Eds.), Proceedings of the workshop on agent simulation: applications, models, and tools (pp. 3–24). Argonne National Laboratory: Argonne, IL.

    Google Scholar 

  • Axtell, R. L., & Epstein, J. M. (1994). Agent based modeling: Understanding our creations. The Bulletin of the Santa Fe Institute, 1994, 28–32.

    Google Scholar 

  • Bigbee, T., Cioffi-Revilla, C., & Luke, S. (2007). Replication of sugarscape using MASON. In T. Terano, H. Kita, H. Deguchi, & K. Kijima (Eds.), Agent-based approaches in economic and social complex systems IV: Post-proceedings of the AESCS international workshop 2005 (pp. 183–190). Tokyo: Springer.

    Chapter  Google Scholar 

  • Bonabeau, E. (2002). Agent-based modeling: Methods and techniques for simulating human systems. Proceedings of the National Academy of Sciences of the United States of America, 99(2), 7280–7287.

    Article  Google Scholar 

  • Castellano, C., Marsili, M., & Vespignani, A. (2000). Nonequilibrium phase transition in a model for social influence. Physical Review Letters, 85(16), 3536–3539.

    Article  Google Scholar 

  • Christley, S., Xiang, X., & Madey, G. (2004). Ontology for agent-based modeling and simulation. In C. M. Macal, D. Sallach, & M. J. North (Eds.), Proceedings of the agent 2004 conference on social dynamics: interaction, reflexivity and emergence. Chicago, IL: Argonne National Laboratory and The University of Chicago. http://www.agent2005.anl.gov/Agent2004.pdf.

  • Cioffi-Revilla, C. (2002). Invariance and universality in social agent-based simulations. Proceedings of the National Academy of Sciences of the United States of America, 99(3), 7314–7316.

    Article  Google Scholar 

  • Conlisk, J. (1996). Why bounded rationality? Journal of Economic Literature, 34(2), 669–700.

    Google Scholar 

  • David, N., Fachada, N., & Rosa, A. C. (2017). Verifying and validating simulations. doi:https://doi.org/10.1007/978-3-319-66948-9_9.

  • Drogoul, A., Vanbergue, D., & Meurisse, T. (2003). Multi-agent based simulation: Where are the agents? In J. S. Sichman, F. Bousquet, & P. Davidsson (Eds.), Proceedings of MABS 2002 multi-agent-based simulation. (Lecture Notes in Computer Science, 2581) (pp. 1–15). Bologna: Springer.

    Google Scholar 

  • Edmonds, B. (2001). The use of models: making MABS actually work. In S. Moss & P. Davidsson (Eds.), Multi-agent-based simulation. (Lecture notes in artificial intelligence, 1979) (pp. 15–32). Berlin: Springer.

    Chapter  Google Scholar 

  • Edmonds, B. (2005). Simulation and complexity: How they can relate. In V. Feldmann & K. Mühlfeld (Eds.), Virtual worlds of precision: Computer-based simulations in the sciences and social sciences (pp. 5–32). Lit-Verlag: Münster.

    Google Scholar 

  • Edmonds, B. (2017). Different modelling purposes. doi:https://doi.org/10.1007/978-3-319-66948-9_4.

  • Edmonds, B., & Hales, D. (2003). Replication, replication and replication: Some hard lessons from model alignment. Journal of Artificial Societies and Social Simulation, 6(4). http://jasss.soc.surrey.ac.uk/6/4/11.html.

  • Edmonds, B., & Hales, D. (2005). Computational Simulation as Theoretical Experiment. Journal of Mathematical Sociology, 29, 1–24.

    Article  Google Scholar 

  • Edwards, M., Huet, S., Goreaud, F., & Deffuant, G. (2003). Comparing an individual-based model of behaviour diffusion with its mean field aggregate approximation. Journal of Artificial Societies and Social Simulation, 6(4). http://jasss.soc.surrey.ac.uk/6/4/9.html.

  • Epstein, J. M. (1999). Agent-based computational models and generative social science. Complexity, 4(5), 41–60.

    Article  MathSciNet  Google Scholar 

  • Epstein, J. M. (2008). Why model?. Journal of Artificial Societies and Social Simulation, 11(4), 12. http://jasss.soc.surrey.ac.uk/11/4/12.html.

  • Epstein, J. M., & Axtell, R. L. (1996). Growing artificial societies: Social science from the bottom up. Cambridge, MA: Brookings Institution Press/MIT Press.

    Google Scholar 

  • Fensel, D. (2001). Ontologies: A silver bullet for knowledge management and electronic commerce. Berlin: Springer.

    Book  MATH  Google Scholar 

  • Galán, J. M., et al. (2009). Errors and artefacts in agent-based modelling. Journal of Artificial Societies and Social Simulation, 12(1). http://jasss.soc.surrey.ac.uk/12/1/1.html.

  • Galán, J. M., & Izquierdo, L. R. (2005). Appearances can be deceiving: lessons learned re-implementing Axelrod’s ‘evolutionary approach to norms’. Journal of Artificial Societies and Social Simulation, 8(3). http://jasss.soc.surrey.ac.uk/8/3/2.html

  • Gilbert, N. (1999). Simulation: A new way of doing social science. The American Behavioral Scientist, 42(10), 1485–1487.

    Google Scholar 

  • Gilbert, N. (2007). Agent-based models. London: Sage Publications.

    Google Scholar 

  • Gilbert, N., & Terna, P. (2000). How to build and use agent-based models in social science. Mind & Society, 1(1), 57–72.

    Article  Google Scholar 

  • Gilbert, N., & Troitzsch, K. G. (1999). Simulation for the social scientist. Buckingham: Open University Press.

    Google Scholar 

  • Gotts, N. M., Polhill, J. G. & Adam, W. J. (2003, 18–21 September). Simulation and analysis in agent-based modelling of land use change. Online proceedings of the first conference of the European Social Simulation Association, Groningen, The Netherlands, http://www.uni-koblenz.de/~essa/ESSA2003/gotts_polhill_adam-rev.pdf.

  • Gruber, T. R. (1993). A translation approach to portable ontology specifications. Knowledge Acquisition, 5(2), 199–220.

    Article  Google Scholar 

  • Hare, M., & Deadman, P. (2004). Further towards a taxonomy of agent-based simulation models in environmental management. Mathematics and Computers in Simulation, 64(1), 25–40.

    Article  MathSciNet  MATH  Google Scholar 

  • Hernández, C. (2004). Herbert A. Simon, 1916-2001, y el Futuro de la Ciencia Económica. Revista Europea De Dirección y Economía De La Empresa, 13(2), 7–23.

    Google Scholar 

  • Heywood, J. G., Masuda, K., Rautmann, R., & Solonnikov, V. A. (Eds.). (1990). The Navier-Stokes equations: Theory and numerical methods; Proceedings of a conference held at Oberwolfach, FRG, Sept. 18–24, 1988. (Lecture Notes in Mathematics, 1431). Berlin: Springer.

    MATH  Google Scholar 

  • Holland, J. H., & Miller, J. H. (1991). Artificial adaptive agents in economic theory. American Economic Review, 81(2), 365–370.

    Google Scholar 

  • Izquierdo, L. R., & Polhill, J. G. (2006). Is your model susceptible to floating point errors? Journal of Artificial Societies and Social Simulation, 9(4). http://jasss.soc.surrey.ac.uk/9/4/4.html.

  • Kleijnen, J. P. C. (1995). Verification and validation of simulation models. European Journal of Operational Research, 82(1), 145–162.

    Article  MathSciNet  MATH  Google Scholar 

  • Kleindorfer, G. B., O'Neill, L., & Ganeshan, R. (1998). Validation in simulation: Various positions in the philosophy of science. Management Science, 44(8), 1087–1099.

    Article  MATH  Google Scholar 

  • Klemm, K., Eguíluz, V., Toral, R., & San Miguel, M. (2003a). Role of dimensionality in Axelrod’s model for the dissemination of culture. Physica A, 327, 1–5.

    Article  MathSciNet  MATH  Google Scholar 

  • Klemm, K., Eguíluz, V., Toral, R., & San Miguel, M. (2003b). Global culture: A noise-induced transition in finite systems. Physical Review E, 67(4), 045101.

    Article  Google Scholar 

  • Klemm, K., Eguíluz, V., Toral, R., & San Miguel, M. (2003c). Nonequilibrium transitions in complex networks: A model of social interaction. Physical Review E, 67(2), 026120.

    Article  Google Scholar 

  • Klemm, K., Eguíluz, V., Toral, R., & San Miguel, M. (2005). Globalization, polarization and cultural drift. Journal of Economic Dynamics & Control, 29(1–2), 321–334.

    Article  MATH  Google Scholar 

  • Kluver, J., & Stoica, C. (2003). Simulations of group dynamics with different models. Journal of Artificial Societies and Social Simulation, 6(4). http://jasss.soc.surrey.ac.uk/6/4/8.html.

  • Leombruni, R., & Richiardi, M. (2005). Why are economists sceptical about agent-based simulations? Physica A, 355, 103–109.

    Article  MathSciNet  Google Scholar 

  • Moss, S. (2001). Game theory: Limitations and an alternative. Journal of Artificial Societies and Social Simulation, 4(2). http://jasss.soc.surrey.ac.uk/4/2/2.html.

  • Moss, S. (2002). Agent based modelling for integrated assessment. Integrated Assessment, 3(1), 63–77.

    Article  Google Scholar 

  • Moss, S., Edmonds, B., & Wallis, S. (1997). Validation and verification of computational models with multiple cognitive agents (Report no. 97–25). Manchester: Centre for Policy Modelling, http://cfpm.org/cpmrep25.html.

  • Ostrom, T. (1988). Computer simulation: The third symbol system. Journal of Experimental Social Psychology, 24(5), 381–392.

    Article  Google Scholar 

  • Parunak, H. V. D., Savit, R., & Riolo, R. L. (1998). Agent-based modeling vs. equation-based modeling: A case study and users’ guide. In J. S. Sichman, R. Conte, & N. Gilbert (Eds.), Multi-agent systems and agent-based simulation. (Lecture notes in artificial intelligence 1534) (pp. 10–25). Berlin: Springer.

    Chapter  Google Scholar 

  • Pavón, J. & Gómez-Sanz, J. (2003). Agent oriented software engineering with INGENIAS. In V. Marik, J. Müller & M. Pechoucek (Eds.), Multi-agent systems and applications III, 3rd international central and eastern European conference on multi-agent systems, CEEMAS. (Lecture notes in artificial intelligence, 2691) (pp. 394–403); Berlin, Heidelberg: Springer.

    Google Scholar 

  • Pignotti, E., Edwards, P., Preece, A., Polhill, J.G. & Gotts, N.M. (2005). Semantic support for computational land-use modelling. Proceedings of the 5th international symposium on cluster computing and the grid (CCGRID 2005) (pp. 840–847). Piscataway, NJ: IEEE Press.

    Google Scholar 

  • Polhill, J. G. & Gotts, N. M. (2006, August 21–25). A new approach to modelling frameworks. Proceedings of the first world congress on social simulation. (Vol. 1, pp. 215–222), Kyoto, Japan.

    Google Scholar 

  • Polhill, J. G., & Izquierdo, L. R. (2005). Lessons learned from converting the artificial stock market to interval arithmetic. Journal of Artificial Societies and Social Simulation, 8(2). http://jasss.soc.surrey.ac.uk/8/2/2.html.

  • Polhill, J. G., Izquierdo, L. R., & Gotts, N. M. (2005). The ghost in the model (and other effects of floating point arithmetic). Journal of Artificial Societies and Social Simulation, 8(1). http://jasss.soc.surrey.ac.uk/8/1/5.html.

  • Polhill, J. G., Izquierdo, L. R., & Gotts, N. M. (2006). What every agent based modeller should know about floating point arithmetic. Environmental Modelling & Software, 21(3), 283–309.

    Article  Google Scholar 

  • Riolo, R. L., Cohen, M. D., & Axelrod, R. M. (2001). Evolution of cooperation without reciprocity. Nature, 411, 441–443.

    Article  Google Scholar 

  • Sakoda, J. M. (1971). The checkerboard model of social interaction. Journal of Mathematical Sociology, 1(1), 119–132.

    Article  Google Scholar 

  • Salvi, R. (2002). The Navier-Stokes equation: Theory and numerical methods. (Lecture notes in pure and applied mathematics). New York: Marcel Dekker.

    Google Scholar 

  • Sansores, C., & Pavón, J. (2005, November 14–18). Agent-based simulation replication: A model driven architecture approach. In A. F. Gelbukh, A. de Albornoz, & H. Terashima-Marín (Eds.), Proceedings of MICAI 2005: Advances in artificial intelligence, 4th Mexican international conference on artificial intelligence. (Lecture notes in computer science, 3789) (pp. 244–253), Monterrey, Mexico. Berlin, Heidelberg: Springer.

    Google Scholar 

  • Sansores, C., Pavón, J., & Gómez-Sanz, J. (2006, July 25). Visual modeling for complex agent-based simulation systems. In J. S. Sichman & L. Antunes (Eds.), Multi-agent-based simulation VI, International workshop, MABS 2005, revised and invited papers. (Lecture notes in computer science, 3891) (pp. 174–189), Utrecht, The Netherlands. Berlin, Heidelberg: Springer.

    Google Scholar 

  • Sargent, R. G. (2003). Verification and validation of simulation models. In S. Chick, P. J. Sánchez, D. Ferrin, & D. J. Morrice (Eds.), Proceedings of the 2003 winter simulation conference (pp. 37–48). Piscataway, NJ: IEEE.

    Google Scholar 

  • Schelling, T. C. (1971). Dynamic models of segregation. Journal of Mathematical Sociology, 1(2), 47–186.

    Article  MATH  Google Scholar 

  • Schelling, T. C. (1978). Micromotives and macrobehavior. New York: Norton.

    Google Scholar 

  • Schmeiser, B. W. (2001, December 09–12). Some myths and common errors in simulation experiments. In B. A. Peters, J. S. Smith, D. J. Medeiros, & M. W. Rohrer (Eds.), Proceedings of the winter simulation conference (Vol. 1, pp. 39–46), Arlington, VA.

    Google Scholar 

  • Takadama, K., Suematsu, Y. L., Sugimoto, N., Nawa, N. E., & Shimohara, K. (2003). Cross-element validation in multiagent-based simulation: Switching learning mechanisms in agents. Journal of Artificial Societies and Social Simulation, 6(4). http://jasss.soc.surrey.ac.uk/6/4/6.html.

  • Taylor, A. J. (1983). The verification of dynamic simulation models. Journal of the Operational Research Society, 34(3), 233–242.

    Article  Google Scholar 

  • Xu, J., Gao, Y. & Madey, G. (2003, April 13–15). A docking experiment: swarm and repast for social network modeling. In Seventh annual swarm researchers conference (SwarmFest 2003. Notre Dame, IN.

    Google Scholar 

  • Yilmaz, L. (2006). Validation and verification of social processes within agent-based computational organization models. Computational & Mathematical Organization Theory, 12(4), 283–312.

    Article  MATH  Google Scholar 

Download references

Acknowledgements

The authors have benefited from the financial support of the Spanish Ministry of Education and Science (projects CSD2010-00034, DPI2004-06590, DPI2005-05676, and TIN2008-06464-C03-02) and of the Junta de Castilla y León (projects BU034A08 and VA006B09). We are also very grateful to Nick Gotts, Gary Polhill, Bruce Edmonds, and Cesáreo Hernández for many discussions on the philosophy of modelling.

Author information

Authors and Affiliations

  1. Department of Civil Engineering, Universidad de Burgos, E-09001, Burgos, Spain

    José M. Galán, Luis R. Izquierdo, José I. Santos & Ricardo del Olmo

  2. Departamento de Organización de Empresas y C.I.M., Universidad de Valladolid, E-47011, Valladolid, Spain

    Segismundo S. Izquierdo & Adolfo López-Paredes

Authors
  1. José M. Galán
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luis R. Izquierdo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Segismundo S. Izquierdo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. José I. Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ricardo del Olmo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Adolfo López-Paredes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to José M. Galán .

Editor information

Editors and Affiliations

  1. Centre for Policy Modelling, Manchester Metropolitan University, Business School, Manchester, United Kingdom

    Bruce Edmonds

  2. Centre for Policy Modelling, Manchester Metropolitan University, Business School, Manchester, United Kingdom

    Ruth Meyer

Further Reading

Further Reading

Gilbert (2007) provides an excellent basic introduction to agent-based modelling. Chapter 4 summarises the different stages involved in an agent-based modelling project, including verification and validation. The paper entitled “Some myths and common errors in simulation experiments” (Schmeiser 2001) discusses briefly some of the most common errors found in simulation from a probabilistic and statistical perspective. The approach is not focused specifically on agent-based modelling but on simulation in general. Yilmaz (2006) presents an analysis of the life cycle of a simulation study and proposes a process-centric perspective for the validation and verification of agent-based computational organisation models. An antecedent of this chapter can be found in Galán et al. (2009). Finally, Chap. 9 in this volume (David et al. 2017) discusses validation in detail.

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Galán, J.M., Izquierdo, L.R., Izquierdo, S.S., Santos, J.I., del Olmo, R., López-Paredes, A. (2017). Checking Simulations: Detecting and Avoiding Errors and Artefacts. In: Edmonds, B., Meyer, R. (eds) Simulating Social Complexity. Understanding Complex Systems. Springer, Cham. https://doi.org/10.1007/978-3-319-66948-9_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-66948-9_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-66947-2

  • Online ISBN: 978-3-319-66948-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation