When the search for truth fails: A computer simulation of the impact of the publication bias on the meta-analysis of scientific literature

Abstract

The production of scientific knowledge is a complex social process, where many actors contribute by their publications to the disclosure of the hidden truth. However, due to different methods, analysed samples, and control variables, empirical findings from this process are often contradictory. Thus, quantitative sciences use meta-analyses in order to extract the likely truth from a corpus of publications about a given research question. Unfortunately, this procedure is often impaired by different forms of the so-called publication bias: papers with null-results are sometimes not published due to the publication policy of journal editors and their boards. Similarly, articles with a high news value may have a better chance of being published, even if their findings finally prove to be wrong. Thus the publications used for meta-analyses are often distorted and lead to wrong conclusions about the truth. For this reason the present article develops a formal model of the effects of the publication bias on the results of meta-analyses. It is successfully tested with empirical data and used for studying the conditions, under which meta-analyses disclose, obscure, or revert the underlying truth. As a main result of the related computer simulations it turns out that the publication bias has for true zero-relations other consequences than for true non-zero relations. Moreover, there are situations where certain forms of the publication bias have unexpectedly favourable effects on the disclosure of the truth by meta-analyses.

Appendix

See Table 1.

Table 1 Number and shares of different types of publications dealing with the correlation r between materialism and wellbeing