Skip to main content
SpringerLink
Log in

Multiversal Methods in Observational Studies: The Case of COVID-19

  • Conference paper
  • First Online:
Studies in Theoretical and Applied Statistics (SIS 2021)

Part of the book series: Springer Proceedings in Mathematics & Statistics ((PROMS,volume 406))

Included in the following conference series:

  • 370 Accesses

Abstract

In the present study, 13 covariates have been selected as potentially associated with 3 metrics of the spread of COVID-19 in 20 European countries. Robustness of the linear correlations between 10 of the 13 covariates as main regressors and the 3 COVID-19 metrics as dependent variables have been tested through a methodology for sensitivity analysis that falls under the name of “Multiverse”. Under this methodology, thousands of alternative estimates are generated by a single hypothesis of regression. The capacity of identification of a robust causal claim for the 10 variables has been measured through 3 indicators over a Janus Confusion Matrix, which is a confusion matrix that assumes the likelihood to observe a True claim as the ratio between the absolute difference of estimates with a different sign and the total of estimates. This methodology provides the opportunity to evaluate the outcomes of a shift from the common level of significance \(\alpha = .05\) to the alternative \(\alpha = .005\). According to the results of the study, in the dataset the benefits of the shifts come at a very high cost in terms of false negatives.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Suggested tutorial: https://dcosme.github.io/specification-curves/SCA_tutorial_inferential.

  2. 2.

    Think about the deep metrological differences between Richter and Mercalli scales in measurement of earthquake magnitude.

  3. 3.

    Janus was the Roman god of gates and was always represented with two faces pointing towards opposite directions, hence the name of the effect.

  4. 4.

    https://www.who.int/.

  5. 5.

    https://www.ecdc.europa.eu/en.

  6. 6.

    UK provides demographic data to Eurostat being a EU member until 2021. Other countries (e.g., Poland) are excluded from the dataset due to missing values.

  7. 7.

    The paper is also part of research line on vulnerability and risk management of the project GRIDAVI Risk Management, Decision Uncertainties and Social Vulnerabilities by the University Research Incentive Plan 2020/2022 called PIACERI.

References

  1. Almagro, M., Orane-Hutchinson, A.: JUE insight: the determinants of the differential exposure to COVID-19 in New York city and their evolution over time. J. Urban Econ. (2020). https://doi.org/10.1016/j.jue.2020.103293

    Article  Google Scholar 

  2. Anderson, S.F., Maxwell, S.E.: Addressing the “replication crisis’’: using original studies to design replication studies with appropriate statistical power. Multivar. Behav. Res. 52(3), 305–324 (2017)

    Google Scholar 

  3. Athey, S., Imbens, G.: A measure of robustness to misspecification. Am. Econ. Rev. 105(5), 476–80 (2015)

    Google Scholar 

  4. Begley, C.G., Ellis, L.M.: Raise standards for preclinical cancer research. Nature 483(7391), 531–533 (2012)

    Google Scholar 

  5. Benjamin, D.J., et al.: Redefine statistical significance. Nat. Hum. Behav. 2(1), 6–10 (2018)

    MathSciNet  Google Scholar 

  6. Bossuyt, P.M.: Laboratory measurement’s contribution to the replication and application crisis in clinical research. Clin. Chem. 65(12), 1479–1480 (2019)

    Google Scholar 

  7. Bruns, S.B., Ioannidis, J.P.: P-curve and p-hacking in observational research. PLoS ONE 11(2), e0149144 (2016)

    Google Scholar 

  8. Busetto, L., et al.: Obesity and COVID-19: an Italian snapshot. Obesity 28(9), 1600–1605 (2020)

    Google Scholar 

  9. Caci, G., et al.: COVID-19 and obesity: dangerous liaisons. J. Clin. Med. 9(8), 2511 (2020)

    Google Scholar 

  10. Camerer, C.F., et al.: Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2(9), 637–644 (2018)

    Google Scholar 

  11. Chang, S., et al.: Mobility network models of COVID-19 explain inequities and inform reopening. Nature 589(7840), 82–87 (2021)

    Google Scholar 

  12. Charaudeau, S., et al.: Commuter mobility and the spread of infectious diseases: application to influenza in France. PLoS ONE 9(1), e83002 (2014)

    Google Scholar 

  13. Chicco, D., Jurman, G.: The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genom. 21(1), 1–13 (2020)

    Google Scholar 

  14. Chu, L., et al.: Vibration of effects in epidemiologic studies of alcohol consumption and breast cancer risk. Int. J. Epidemiol. 49(2), 608–618 (2020)

    Google Scholar 

  15. Conticini, E., et al.: Can atmospheric pollution be considered a co-factor in extremely high level of SARS-CoV-2 lethality in Northern Italy? Environ. Pollut. 261,(2020). https://doi.org/10.1016/j.envpol.2020.114465

  16. Cosme, D., et al.: Multivariate neural signatures for health neuroscience: assessing spontaneous regulation during food choice. Soc. Cogn. Affect. Neurosci. 15(10), 1120–1134 (2020)

    Google Scholar 

  17. Cui, Y., et al.: Air pollution and case fatality of SARS in the People’s Republic of China: an ecologic study. Environ. Health 2(1), 1–5 (2003)

    MathSciNet  Google Scholar 

  18. Del Giudice, M., Gangestad, S.W.: A traveler’s guide to the multiverse: promises, pitfalls, and a framework for the evaluation of analytic decisions. Adv. Methods Pract. Psychol. Sci. 4(1) (2021). https://doi.org/10.1177/2515245920954925

  19. Donzelli, G., et al.: Relations between air quality and COVID-19 lockdown measures in Valencia, Spain. Int. J. Environ. Res. Public Health 18(5), 2296 (2021)

    Google Scholar 

  20. Duvendack, M., et al.: What is meant by “replication’’ and why does it encounter resistance in economics? Am. Econ. Rev. 107(5), 46–51 (2017)

    Google Scholar 

  21. Earp, B.D., Trafimow, D.: Replication, falsification, and the crisis of confidence in social psychology. Front. Psychol. (2015). https://doi.org/10.3389/fpsyg.2015.00621

    Article  Google Scholar 

  22. Ebinger, J.E., et al.: Pre-existing traits associated with Covid-19 illness severity. PLoS ONE 15(7), e0236240 (2020)

    Google Scholar 

  23. Espejo-Paeres, C., Núñez-Gil, I.J., Estrada, V., et al.: Impact of smoking on COVID-19 outcomes: a HOPE Registry subanalysis. BMJ Nutr. Prev. Health 4, (2021). https://doi.org/10.1136/bmjnph-2021-000269

  24. European Environmental Agency: Air Quality in Europe—2020 report, 9/2020, EEA Report (2020)

    Google Scholar 

  25. Eurostat: Eurostat Regional Yearbook, Edition 2020 (2020)

    Google Scholar 

  26. Fadini, G.P., et al.: Newly-diagnosed diabetes and admission hyperglycemia predict COVID-19 severity by aggravating respiratory deterioration. Diabetes Res. Clin. Pract. 168 (2020). https://doi.org/10.1016/j.diabres.2020.108374

  27. Gauchat, G.: Politicization of science in the public sphere: a study of public trust in the United States, 1974 to 2010. Am. Sociol. Rev. 77(2), 167–187 (2012)

    Google Scholar 

  28. Gelman, A., Loken, E.: The statistical crisis in science. Data-dependent analysis—a “garden of forking paths”—explains why many statistically significant comparisons don’t hold up. Am. Sci. 102(6), 460–466 (2014)

    Google Scholar 

  29. Granger, C.W., Uhlig, H.F.: Reasonable extreme-bounds analysis. J. Econometrics 44(1–2), 159–170 (1990)

    MathSciNet  MATH  Google Scholar 

  30. Halsey, L.G., et al.: The fickle P value generates irreproducible results. Nat. Methods 12(3), 179–185 (2015)

    Google Scholar 

  31. Hamidi, S., et al.: Does density aggravate the COVID-19 pandemic? Early findings and lessons for planners. J. Am. Plann. Assoc. 86(4), 495–509 (2020)

    Google Scholar 

  32. Harder, J.A.: The multiverse of methods: extending the multiverse analysis to address data-collection decisions. Perspect. Psychol. Sci. 15(5), 1158–1177 (2020)

    Google Scholar 

  33. Head, M.L., et al.: The extent and consequences of p-hacking in science. PLoS Biol. 13(3) (2015). https://doi.org/10.1371/journal.pbio.1002106

  34. Hicks, D.J.: Open science, the replication crisis, and environmental public health. Accountability Res. 1-29 (2021). https://doi.org/10.1080/08989621.2021.1962713

  35. Imbens, G.W.: Statistical significance, p-values, and the reporting of uncertainty. J. Econ. Perspect. 35(3), 157–74 (2021)

    MathSciNet  Google Scholar 

  36. Ioannidis, J.P., et al.: Increasing value and reducing waste in research design, conduct, and analysis. The Lancet 383(9912), 166–175 (2014)

    Google Scholar 

  37. Ioannidis, J.P.: The proposal to lower P value thresholds to 0.005. JAMA 319(14), 1429–1430 (2018)

    Google Scholar 

  38. Ioannidis, J.P.: What have we (not) learnt from millions of scientific papers with P values? Am. Stat. 73(sup1), 20–25 (2019)

    MathSciNet  MATH  Google Scholar 

  39. Islam, T.U., Rizwan, M.: Comparison of correlation measures for nominal data. Commun. Stat. Simul. Comput. 1–20 (2020). https://doi.org/10.1080/03610918.2020.1869984

  40. ISS EpiCentro: I dati per l’Italia. La Sorveglianza Passi d’Argento (2020). https://www.epicentro.iss.it/passi-argento

  41. Jewell, N.P., et al.: Predictive mathematical models of the COVID-19 pandemic: underlying principles and value of projections. JAMA 323(19), 1893–1894 (2020)

    Google Scholar 

  42. Johnson, V.E.: Revised standards for statistical evidence. Proc. Natl. Acad. Sci. 110(48), 19313–19317 (2013)

    MATH  Google Scholar 

  43. Klau, S., et al.: Sampling uncertainty versus method uncertainty: a general framework with applications to omics biomarker selection. Biometrical J. 62(3), 670–687 (2020)

    MathSciNet  MATH  Google Scholar 

  44. Kogevinas, M., et al.: Ambient air pollution in relation to SARS-CoV-2 infection, antibody response, and COVID-19 disease: a cohort study in Catalonia, Spain (COVICAT study). Environ. Health Perspect. 129(11) (2021). https://doi.org/10.1289/EHP9726

  45. Kreps, S.E., Kriner, D.L.: Model uncertainty, political contestation, and public trust in science: evidence from the COVID-19 pandemic. Sci. Adv. 6(43) (2020). https://doi.org/10.1126/sciadv.abd4563

  46. Leamer, E.E.: Sensitivity analyses would help. Am. Econ. Rev. 75(3), 308–313 (1985)

    Google Scholar 

  47. Lee, Y.J.: The impact of the COVID-19 pandemic on vulnerable older adults in the United States. J. Gerontol. Soc. Work 63(6–7), 559–564 (2020)

    Google Scholar 

  48. Leek, J.T., Peng, R.D.: Statistics: P values are just the tip of the iceberg. Nat. News 520(7549), 612 (2015)

    Google Scholar 

  49. Mao, Z., et al.: Investigating the self-reported health status of domestic and overseas Chinese populations during the COVID-19 pandemic. Int. J. Environ. Res. Public Health 18(6), 3043 (2021)

    Google Scholar 

  50. Masur, P.K.: Understanding the effects of conceptual and analytical choices on ‘finding’ the privacy paradox: a specification curve analysis of large-scale survey data. Inf. Commun. Soc. 1–19 (2021). https://doi.org/10.1080/1369118X.2021.1963460

  51. Masur, P.K., Scharkow, M.: specr: Conducting and Visualizing Specification Curve Analyses. R Package (2020)

    Google Scholar 

  52. Maxwell, S.E., et al.: Is psychology suffering from a replication crisis? What does “failure to replicate’’ really mean? Am. Psychol. 70(6), 487–498 (2015)

    Google Scholar 

  53. Mayo, D.G., Spanos, A.: Methodology in practice: statistical misspecification testing. Philos. Sci. 71(5), 1007–1025 (2004)

    MathSciNet  Google Scholar 

  54. Mazzola, J.J., Deuling, J.K.: Forgetting what we learned as graduate students: HARKing and selective outcome reporting in I-O journal articles. Ind. Organ. Psychol. 6(3), 279–284 (2013)

    Google Scholar 

  55. McShane, B.B., et al.: Abandon statistical significance. Am. Stat. 73(sup1), 235–245 (2019)

    MathSciNet  MATH  Google Scholar 

  56. Muñoz, J., Young, C.: We ran 9 billion regressions: eliminating false positives through computational model robustness. Sociol. Methodol. 48(1), 1–33 (2018)

    Google Scholar 

  57. Nižetić, S.: Impact of coronavirus (COVID-19) pandemic on air transport mobility, energy, and environment: a case study. Int. J. Energy Res. 44(13), 10953–10961 (2020)

    Google Scholar 

  58. Open Science Collaboration: Estimating the reproducibility of psychological science. Science 349(6251) (2015). https://doi.org/10.1126/science.aac4716

  59. Orben, A., Przybylski, A.K.: The association between adolescent well-being and digital technology use. Nat. Hum. Behav. 3(2), 173–182 (2019)

    Google Scholar 

  60. Oztig, L.I., Askin, O.E.: Human mobility and coronavirus disease 2019 (COVID-19): a negative binomial regression analysis. Public Health 185, 364–367 (2020). https://doi.org/10.1016/j.puhe.2020.07.002

  61. Page, L., et al.: The replication crisis, the rise of new research practices and what it means for experimental economics. J. Econ. Sci. Assoc. 1–16 (2021). https://doi.org/10.1007/s40881-021-00107-7

  62. Palpacuer, C., et al.: Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis. BMC Med. 17(1), 1–13 (2019)

    Google Scholar 

  63. Pansini, R., Fornacca, D.: COVID-19 higher mortality in Chinese Regions with chronic exposure to lower air quality. Front. Public Health 8, (2021). https://doi.org/10.3389/fpubh.2020.597753

  64. Parohan, M., et al.: Risk factors for mortality in patients with Coronavirus disease 2019 (COVID-19) infection: a systematic review and meta-analysis of observational studies. The Aging Male 23(5), 1416–1424 (2020)

    Google Scholar 

  65. Patel, C.J., et al.: Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. J. Clin. Epidemiol. 68(9), 1046–1058 (2015)

    Google Scholar 

  66. Perone, G.: The determinants of COVID-19 case fatality rate (CFR) in the Italian regions and provinces: an analysis of environmental, demographic, and healthcare factors. Sci. Total Environ. 760, (2021). https://doi.org/10.1016/j.scitotenv.2020.142523

  67. Pike, H.: Statistical significance should be abandoned, say scientists. BMJ 364, (2019). https://doi.org/10.1136/bmj.l1374

  68. Pluchino, A., et al.: A novel methodology for epidemic risk assessment of COVID-19 outbreak. Sci. Rep. 11(1), 1–20 (2021)

    Google Scholar 

  69. Qiu, F., et al.: Impacts of cigarette smoking on immune responsiveness: up and down or upside down? Oncotarget 8(1), 268–284 (2017)

    Google Scholar 

  70. Rohrer, J.M., et al.: Probing birth-order effects on narrow traits using specification-curve analysis. Psychol. Sci. 28(12), 1821–1832 (2017)

    Google Scholar 

  71. Sala-i-Martin, X.: I just ran four million regressions. Am. Econ. Rev. 87(2), 178–183 (1997)

    Google Scholar 

  72. Saraceno, J., et al.: Reevaluating the substantive representation of lesbian, gay, and bisexual Americans: a multiverse analysis. J. Politics 83(4), 1837–1843 (2021)

    Google Scholar 

  73. Schmeiser, H., et al.: The risk of model misspecification and its impact on solvency measurement in the insurance sector. J. Risk Finance 13(4), 285–308 (2012)

    Google Scholar 

  74. Seitshiro, M.B., Mashele, H.P.: Quantification of model risk that is caused by model misspecification. J. Appl. Stat. 1–21 (2020). https://doi.org/10.1080/02664763.2020.1849055

  75. Setti, L., et al.: Potential role of particulate matter in the spreading of COVID-19 in Northern Italy: first observational study based on initial epidemic diffusion. BMJ Open 10(9), e039338 (2020)

    Google Scholar 

  76. Sharifi, A., Khavarian-Garmsir, A.R.: The COVID-19 pandemic: impacts on cities and major lessons for urban planning, design, and management. Sci. Total Environ. (2020). https://doi.org/10.1016/j.scitotenv.2020.142391

    Article  Google Scholar 

  77. Simmons, J.P., et al.: False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22(11), 1359–1366 (2011)

    Google Scholar 

  78. Simonsohn, U., et al.: Specification curve analysis. Nat. Hum. Behav. 4(11), 1208–1214 (2020)

    Google Scholar 

  79. Simonsohn, U., et al.: Specification curve: Descriptive and inferential statistics on all reasonable specifications. Available at:SSRN (2019). https://doi.org/10.2139/ssrn.2694998

  80. Sönning, L., Werner, V.: The replication crisis, scientific revolutions, and linguistics. Linguistics 59(5), 1179–1206 (2021)

    Google Scholar 

  81. Steegen, S., et al.: Increasing transparency through a multiverse analysis. Perspect. Psychol. Sci. 11(5), 702–712 (2016)

    Google Scholar 

  82. Taroni, F., et al.: Statistical hypothesis testing and common misinterpretations: should we abandon p-value in forensic science applications? Forensic Sci. Int. 259, e32–e36 (2016)

    Google Scholar 

  83. Trafimow, D.: Five nonobvious changes in editorial practice for editors and reviewers to consider when evaluating submissions in a post p\(<\) 0.05 universe. Am. Stat. 73(sup1), 340–345 (2019)

    Google Scholar 

  84. Trafimow, D., et al.: Manipulating the alpha level cannot cure significance testing. Front. Psychol. 9, (2018). https://doi.org/10.3389/fpsyg.2018.00699

  85. Vanpaemel, W., et al.: Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra 1(1), 1–5 (2015)

    Google Scholar 

  86. Warrens, M.J.: On association coefficients for 2\(\times \) 2 tables and properties that do not depend on the marginal distributions. Psychometrika 73(4), 777 (2008)

    MathSciNet  MATH  Google Scholar 

  87. Wasserstein, R.L., Lazar, N.A.: The ASA statement on p-values: context, process, and purpose. Am. Stat. 70(2), 129–133 (2016)

    MathSciNet  Google Scholar 

  88. Wasserstein, R.L. et al.: Moving to a world beyond “p\(<\) 0.05”. Am. Stat. 73(sup. 1), 1–19 (2019)

    Google Scholar 

  89. Wei, E.K., et al.: Nine lessons learned from the COVID-19 pandemic for improving hospital care and health care delivery. JAMA Intern. Med. 181(9), 1161–1163 (2021)

    Google Scholar 

  90. World Health Organization: Coronavirus Disease 2019 (COVID-19) (2021). Available at: https://apps.who.int/iris/handle/10665/331475

  91. Yaffe, J.: From the editor—do we have a replication crisis in social work research? J. Soc. Work Educ. 55(1), 1–4 (2019)

    MathSciNet  Google Scholar 

  92. Young, C., Holsteen, K.: Model uncertainty and robustness: a computational framework for multimodel analysis. Sociol. Methods Res. 46(1), 3–40 (2017)

    MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Political and Social Sciences, University of Catania, 8, Vittorio Emanuele II, 95131, Catania, Italy

    Venera Tomaselli

  2. Department of Physics and Astronomy “E. Majorana”, University of Catania, 64, S. Sofia, 95123, Catania, Italy

    Giulio Giacomo Cantone & Vincenzo Miracula

Authors
  1. Venera Tomaselli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giulio Giacomo Cantone
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vincenzo Miracula
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Venera Tomaselli .

Editor information

Editors and Affiliations

  1. Department of Economics and Management, University of Pisa, Pisa, Italy

    Nicola Salvati

  2. Department of Economics and Statistics, University of Salerno, Fisciano, Salerno, Italy

    Cira Perna

  3. Department of Economics and Management, University of Pisa, Pisa, Italy

    Stefano Marchetti

  4. School of Mathematics and Applied Statistics, University of Wollongong, Wollongong, NSW, Australia

    Raymond Chambers

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tomaselli, V., Cantone, G.G., Miracula, V. (2022). Multiversal Methods in Observational Studies: The Case of COVID-19. In: Salvati, N., Perna, C., Marchetti, S., Chambers, R. (eds) Studies in Theoretical and Applied Statistics . SIS 2021. Springer Proceedings in Mathematics & Statistics, vol 406. Springer, Cham. https://doi.org/10.1007/978-3-031-16609-9_22

Download citation

Publish with us

Policies and ethics

Search

Navigation