Skip to main content
SpringerLink
Log in

Searching for explanations: testing social scientific methods in synthetic ground-truthed worlds

  • S.I. : Ground Truth: in silico Social Science (GTIS3)
  • Published:
Computational and Mathematical Organization Theory Aims and scope Submit manuscript

Abstract

A scientific model’s usefulness relies on its ability to explain phenomena, predict how such phenomena will be impacted by future interventions, and prescribe actions to achieve desired outcomes. We study methods for learning causal models that explain the behaviors of simulated “human” populations. Through the Ground Truth project, we solved a series of Challenges where our explanations, predictions and prescriptions were scored against ground truth information. We describe the processes that emerged for applying causal discovery, network analysis, agent-based modeling and other analytical methods to inform solutions to Challenge tasks. We present our team’s overall performance results on these Challenges and discuss implications for future efforts to validate social scientific research using simulation-based challenges.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. ananke is shared at: https://gitlab.com/causal/ananke with documentation available at: https://ananke.readthedocs.io/en/latest/.

  2. The graphml format is documented at http://graphml.graphdrawing.org/.

  3. dworp is an open-source project available at https://pypi.org/project/dworp/.

References

  • Bhattacharya R, Malinsky D, Shpitser I (2019a) Causal inference under interference and network uncertainty. In: The 35th conference on uncertainty in artificial intelligence (UAI-19), AUAI Press

  • Bhattacharya R, Nabi R, Shpitser I, Robins JM (2019b) Identification in missing data models represented by directed acyclic graphs. In: The 35th conference on uncertainty in artificial intelligence (UAI-19), AUAI Press

  • Bhattacharya R, Nabi R, Shpitser I (2020) Semiparametric inference for causal effects in graphical models with hidden variables. arXiv:2003.12659 [stat.ML]

  • Dorie V, Hill J, Shalit U, Scott M, Cervone D (2019) Automated versus do-it-yourself methods for causal inference: lessons learned from a data analysis competition. Stat Sci 34(1):43–68

    Article  Google Scholar 

  • Fink C, Schmidt A, Barash V, Kelly J, Cameron CJ, Macy M (2016) Investigating the observability of complex contagion in empirical social networks. In: The international AAAI conference on web and social media (ICWSM), pp 335–348

  • Finkelstein N, Shpitser I (2020) Deriving bounds and inequality constraints using logical relations among counterfactuals. In: The 36th conference on uncertainty in artificial intelligence (UAI-20), AUAI Press

  • Glymour C, Zhang K, Spirtes P (2019) Review of causal discovery methods based on graphical models. Front Genet 10:524

    Article  Google Scholar 

  • Karavani E, El-Hay T, Shimoni Y, Yanover C et al (2019) Comment: causal inference competitions: where should we aim? Stat Sci 34(1):86–89

    Article  Google Scholar 

  • Laatabi A, Marilleau N, Nguyen-Huu T, Hbid H, Babram MA (2018) ODD+ 2D: an ODD based protocol for mapping data to empirical ABMS. J Artif Soc Soc Simul 21(2):9

    Article  Google Scholar 

  • Lee JJR, Shpitser I (2020) Identification methods with arbitrary interventional distributions as inputs. arXiv preprint, arXiv:200401157

  • Malinsky D, Danks D (2018) Causal discovery algorithms: a practical guide. Philos Compass 13(1):e12470

    Article  Google Scholar 

  • Malinsky D, Spirtes P (2018) Causal structure learning from multivariate time series in settings with unmeasured confounding. In: Proceedings of 2018 ACM SIGKDD workshop on causal discovery, pp 23–47

  • Müller B, Bohn F, Dreßler G, Groeneveld J, Klassert C, Martin R, Schlüter M, Schulze J, Weise H, Schwarz N (2013) Describing human decisions in agent-based models-ODD+ D, an extension of the odd protocol. Environ Model Softw 48:37–48

    Article  Google Scholar 

  • Nabi R, Bhattacharya R, Shpitser I (2020) Full law identification in graphical models of missing data: Completeness results. In: The thirty-seventh international conference on machine learning (ICML 2020), arXiv:2004.04872 [stat.ME]

  • Naugle A, Krofcheck D, Warrender C, Lakkaraju K, Swiler L, Verzi S, Emery B, Murdock J, Bernard M, Romero V (2020a) Results of the ground truth program: What can simulation test beds teach us about social science? Comput Math Organ Theory

  • Naugle A, Russell A, Lakkaraju K, Swiler L, Verzi S, Romero V (2020b) The ground truth program: simulations as test beds for social science research methods. Comput Math Organ Theory

  • Ogarrio JM, Spirtes P, Ramsey J (2016) A hybrid causal search algorithm for latent variable models. In: Conference on probabilistic graphical models, pp 368–379

  • Ogburn B, Lee Y, Shpitser I (2018) Causal inference, social networks, and chain graphs. J R Stat Soc A (to appear), arXiv:1812.04990 [stat.ME]

  • Parunak HVD (2020) SCAMP’s stigmergic model of social conflict. Comput Math Organ Theory

  • Pynadath DV, Dilkina B, Jeong DC, John RS, Marsella SC, Merchant C, Miller LC, Read SJ (2020) Disaster world: Decision-theoretic agents for simulating population responses to hurricanes. Comput Math Organ Theory

  • Rager S, Leung A, Pinegar S, Mangels J, Poole MS, Contractor N (2020) Groups, governance, and greed: The access world model. Comput Math Organ Theory

  • Richardson TS, Evans RJ, Robins JM, Shpitser I (2017) Nested markov properties for acyclic directed mixed graphs. In: The thirty-seventh international conference on machine learning (ICML 2020), arXiv:1701.06686 [stat.ME]

  • Robins J (1986) A new approach to causal inference in mortality studies with a sustained exposure period-application to control of the healthy worker survivor effect. Math Model 7(9–12):1393–1512

    Article  Google Scholar 

  • Shalizi CR, Thomas AC (2011) Homophily and contagion are generically confounded in observational social network studies. Sociological methods & research 40(2):211–239. https://doi.org/10.1177/0049124111404820

    Article  Google Scholar 

  • Sherman E, Shpitser I (2018) Identification and estimation of causal effects from dependent data. In: The 32nd annual conference on neural information processing systems (NeurIPS-18), AUAI Press

  • Sherman E, Shpitser I (2019) Intervening on network ties. In: The 35th conference on uncertainty in artificial intelligence (UAI-19), AUAI Press

  • Sherman E, Arbour D, Shpitser I (2020) General identification of dynamic treatment regimes under interference. In: Chiappa S, Calandra R (eds) the 23rd international conference on artificial intelligence and statistics (AISTATS), PMLR, proceedings of machine learning research, vol 108, pp 3917–3927

  • Shpitser I (2015) Segregated graphs and marginals of chain graph models. In: Advances in neural information processing systems, pp 1720–1728

  • Shpitser I, Pearl J (2008) Complete identification methods for the causal hierarchy. J Mach Learn Res 9(Sep):1941–1979

  • Shpitser I, Evans RJ, Richardson TS (2018) Acyclic linear sems obey the nested markov property. In: Conference on uncertainty in artificial intelligence, p 255

  • Spirtes P, Scheines R, Ramsey J, Glymour C (2009) Repository for the Tetrad Project. https://github.com/cmu-phil/tetrad, (version 6.7.0, Accessed 23 Sept 2019)

  • Tchetgen Tchetgen E, Fulcher I, Shpitser I (2017) Auto-g-computation of causal effects on a network. J Am Stat Assoc (to appear), arXiv:1709.01577 [stat.ME]

  • Tikka S, Karvanen J (2017) Identifying causal effects with the R package causaleffect. J Stat Softw 76(12):1–30, https://doi.org/10.18637/jss.v076.i12

  • Verma T, Pearl J (1990) Equivalence and synthesis of causal models [technical report r-150]. University of California, Los Angeles, Department of Computer Science

    Google Scholar 

  • Züfle A, Wenk C, Pfoser D, Crooks A, Kavak H, Kim JS, Jin H (2020) Urban life: a model of people and places. Comput Math Organ Theory

Download references

Acknowledgements

This project is sponsored by the Defense Advanced Research Projects Agency (DARPA) under contract HR0011-18-C-0049. The content of the information does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Author information

Authors and Affiliations

  1. Johns Hopkins University Applied Physics Laboratory (JHU/APL), 11100 Johns Hopkins Road, Laurel, MD, 20723, USA

    Aurora C. Schmidt, Corey Lowman, Joshua Brulé, Ariel M. Greenberg, Cash J. Costello, Alexander Perrone, Yanni A. Kouskoulas & Clay Fink

  2. Cornell University, 372 Uris Hall Cornell University, Ithaca, NY, 14853, USA

    Christopher J. Cameron & Michael W. Macy

  3. Graphika, Inc., 311 W 43rd St., New York, NY, 10036, USA

    Amruta J. Deshpande, Vladimir Barash & Liz McQuillan

  4. University of Hawaii, 2540 Dole Street, Holmes Hall 483, Honolulu, HI, 96822, USA

    Seyyed A. Fatemi & June Zhang

  5. Johns Hopkins University, 3400 N Charles St, Baltimore, MD, 21218, USA

    Eli S. Sherman, Rohit Bhattacharya & Ilya Shpitser

Authors
  1. Aurora C. Schmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christopher J. Cameron
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Corey Lowman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joshua Brulé
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Amruta J. Deshpande
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Seyyed A. Fatemi
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Vladimir Barash
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ariel M. Greenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Cash J. Costello
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Eli S. Sherman
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Rohit Bhattacharya
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Liz McQuillan
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Alexander Perrone
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Yanni A. Kouskoulas
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Clay Fink
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. June Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Ilya Shpitser
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Michael W. Macy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aurora C. Schmidt.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schmidt, A.C., Cameron, C.J., Lowman, C. et al. Searching for explanations: testing social scientific methods in synthetic ground-truthed worlds. Comput Math Organ Theory 29, 156–187 (2023). https://doi.org/10.1007/s10588-021-09353-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10588-021-09353-w

Keywords

Search

Navigation