Skip to main content
SpringerLink
Log in

Theorie und Methoden multipler statistischer Vergleiche

Ein Rückblick auf 80 Jahre multiples Testen

Theory and methods of multiple comparisons: a review of 80 years of multiple testing

  • Originalveröffentlichung
  • Published:
AStA Wirtschafts- und Sozialstatistisches Archiv Aims and scope Submit manuscript

Zusammenfassung

Wird anhand eines Datensatzes simultan über mehrere Nullhy pothesen entschieden, so kann es zu einer Inflation des Fehlers 1. Art kommen. Daher wurden, beginnend mit Fisher (1935), für solche multiplen Hypothesenprobleme spe zielle Tests entwickelt. Trotz ihrer Relevanz für verschiedenste Forschungsbereiche, insbesondere auch für dieWirtschafts- und Sozialwissenschaften, werden solche Ver fahren vergleichsweise selten eingesetzt. Deshalb rekapituliert und systematisiert der vorliegende Aufsatz die Entwicklung der Theorie und Methoden multipler Vergleiche in den 80 Jahren seit Fisher – und verfolgt damit insbesondere das Ziel, Anwender der Statistik hinsichtlich der Problematik des multiplen Testens zu sensibilisieren.

Abstract

As deciding on more than one null hypothesis based upon the same data set can provoke an inflation of the type I error rate, special methods for these multiple testing problems have been developed since Fisher (1935). Although highly relevant for different areas of research, especially for economics and the social sciences, multiple tests are relatively rarely applied. This paper, therefore, reviews and systematizes the evolution of theory and methods concerning multiple comparisons. We particu larly pursue the objective to sensitize users of statistics to the issues related to multiple testing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Abb. 1

Literatur

  • Begun JM, Gabriel KR (1981) Closure of the Newman-Keuls multiple comparisons procedure. J Am Stat Assoc 76:241–245

  • Benjamini Y (2010a) Discovering the false discovery rate. J R Stat Soc B 72:405–416

  • Benjamini Y (2010b) Simultaneous and selective inference: current successes and future challenges. Biom J 52:708–721

  • Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B 57:289–300

  • Benjamini Y, Hochberg Y (1997) Multiple hypotheses testing with weights. Scand J Stat 24:407–418

  • Benjamini Y, Hochberg Y (2000) On the adaptive control of the false discovery rate in multiple testing with independent statistics. J Educ Behav Stat 25:60–83

  • Benjamini Y, Liu W (1999) A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. J Stat Plan Inference 82:163–170

  • Benjamini Y, Yekutieli D (2001) The control of the false discovery rate in multiple testing under dependency. Ann Stat 29:1165–1188

  • Benjamini Y, Krieger AM, Yekutieli D (2006) Adaptive linear step-up procedures that control the false discovery rate. Biometrika 93:491–507

  • Berry DA, Hochberg Y (1999) Bayesian perspectives on multiple comparisons. J Stat Plan Inference 82:215–277

  • Blanchard G, Roquain E (2008) Two simple sufficient conditions for FDR control. Electron J Stat 2:963–992

  • Bodnar T, Dickhaus T (2013) False discovery rate control under Archimedean copula. Arbeitspapier, Institut für Statistik, Humboldt-Universität, Berlin

  • Bohrer R (1979) Multiple three-decision rules for parametric signs. J Am Stat Assoc 74:432–437

  • Bretz F, Hothorn T, Westfall P (2008) Multiple comparison procedures in linear models. In: Brito P (Hrsg) Compstat 2008: Proceedings in Computational Statistics. Physica-Verlag, Heidelberg, S 423–431

  • Bretz F, Hothorn T, Westfall P (2011) Multiple Comparisons Using R. Chapman & Hall/CRC, Boca Raton

  • Cheng PH, Meng YK (1995) A new formula for tail probabilities of Dunnet’s t with unequal sample sizes. Commun Stat 24:523–532

  • Dickhaus T (2014) Simultaneous Statistical Inference with Applications in the Life Sciences. Springer-Verlag, Berlin

  • Dixon DO, Duncan DB (1975) Minimum Bayes risk t-intervals for multiple comparisons. J Am Stat Assoc 70:822–831

  • Dmitrienko A, Tamhane AC (2011) Mixtures of multiple testing procedures for gatekeeping applications in clinical trials. Stat Med 30:1473–1488

  • Dmitrienko A, Tamhane AC, Bretz F (2010) Multiple Testing Problems in Pharmaceutical Statistics. Chapman & Hall/CRC, Boca Raton

  • Dudoit S, van der Laan MJ (2008) Multiple Testing Procedures with Applications to Genomics. Springer-Verlag, New York

  • Dudoit S, Shaffer JP, Boldrick JC (2003) Multiple hypothesis testing in microarray experiments. Stat Sci 18:71–103

  • Dudoit S, van der Laan MJ, Pollard KS (2004) Multiple testing. Part I. Single-step procedures for control of general type one error rates. Stat Appl Genet Mol Biol 3:Article 13

  • Duncan DB (1961) Bayes rules for a common multiple comparisons problem and related student-t problems. Ann Math Stat 32:1013–1033

  • Duncan DB (1965) A Bayesian approach to multiple comparisons. Technometrics 7:171–222

  • Duncan DB (1975) T tests and intervals for comparisons suggested by the data. Biometrics 31:339–359

  • Duncan DB, Godbold JH (1979) Approximate k-ratio t tests for differences between unequally replicated treatments. Biometrics 35:749–756

  • Dunnet CW (1955) A multiple comparison procedure for comparing several treatments with a control. J Am Stat Assoc 50:1096–1121

  • Dunnet CW, Tamhane AC (1992) A step-up multiple test procedure. J Am Stat Assoc 87:162–170

  • Edwards DG, Hsu JC (1983) Multiple comparisons with the best treatment. J Am Stat Assoc 78:965–971

  • Einot I, Gabriel KR (1975) A study of the powers of several methods of multiple comparisons. J Am Stat Assoc 70:574–583

  • Fahrmeir L, Künstler R, Pigeot I (2011) Statistik: Der Weg zur Datenanalyse, 7. Aufl. Springer-Verlag, Berlin

  • Farcomeni A (2008) A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion. Stat Method Med Res 17:347–388

  • Finner H, Strassburger K (2002) The partitioning principle: a powerful tool in multiple decision theory. Ann Stat 30:1194–1213

  • Finner H, Dickhaus T, Roters M (2009) On the false discovery rate and an asymptotically optimal rejection curve. Ann Stat 37:596–618

  • Fisher RA (1935) The Design of Experiments. Oliver and Boyd, Edinburgh und London, aktuell: 9. Aufl. 1974. Hafner, New York

  • Gabriel KR (1969) Simultaneous test procedures: some theory of multiple comparisons. J Am Stat Assoc 40:224–250

  • Gabriel KR (1978) A simple method of multiple comparisons of means. Ann Math Stat 73:724–729

  • Gao X, Alvo M, Chen J, Li G (2008) Nonparametric multiple comparison procedures for unbalanced one-way factorial designs. J Stat Plan Inference 138:2574–2591

  • Goeman JJ, Solari A (2014) Multiple hypothesis testing in genomics. Stat Med 33:1946–1978

  • Hochberg Y (1974) Some conservative generalizations of the T-method in simultaneous inference. J Multivar Anal 4:224–232

  • Hochberg Y (1988) A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75:800–802

  • Hochberg Y, Tamhane AC (1987) Multiple Comparison Procedures. John Wiley & Sons, New York

  • Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6:65–70

  • Hommel G (1988) A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75:383–386

  • Hoover DR (1991) Simultaneous comparisons of multiple treatments to two (or more) controls. Biom J 33:913–921

  • Horn M, Vollandt R (1995) Multiple Tests und Auswahlverfahren. Fischer-Verlag, Stuttgart

  • Hothorn T, Bretz F, Westfall P (2008) Simultaneous inference in general parametric models. Biom J 50:346–363

  • Hsu JC (1984) Ranking and selection and multiple comparisons with the best. In: Santner TJ, Tamhane AC (Hrsg) Design of Experiments: Ranking and Selection. Marcel Dekker, New York

  • Hsu JC (1996) Multiple Comparisons: Theory and Methods. Chapman & Hall, London

  • Konietschke F, Hothorn LA, Brunner E (2012) Rank-based multiple test procedures and simultaneous confidence intervals. Electron J Stat 6:738–759

  • Kramer CY (1956) Extension of multiple range tests to group means with unequal numbers of replications. Biometrics 12:307–310

  • van der Laan MJ, Dudoit S, Pollard KS (2004) Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Stat Appl Genet Mol Biol 1:Article 15

  • van der Laan MJ, Birkner MD, Hubbard AE (2005) Empirical Bayes and resampling-based multiple testing procedure controlling tail probability of the proportion of false positives. Stat Appl Genet Mol Biol 4:Article 29

  • Lehmann EL (1957a) A theory of some multiple decision problems 1. Ann Math Stat 28:1–25

  • Lehmann EL (1957b) A theory of some multiple decision problems 2. Ann Math Stat 28:547–572

  • Lehmann EL, Romano JP (2005a) Generalizations of the familywise error rate. Ann Stat 33:1138–1154

  • Lehmann EL, Romano JP (2005b) Testing Statistical Hypotheses. Springer-Verlag, New York

  • Marcus R, Peritz E, Gabriel KR (1976) On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63:655–660

  • Maurer W, Mellein B (1987) On new multiple tests based on independent p-values and the assessment of their power. In: Bauer P, Hommel G, Sonnemann E (Hrsg) Multiple Hypotheses Testing, Springer-Verlag, Berlin, S 48–66

  • Maurer W, Hothorn LA, Lehmacher W (1995) Multiple comparisons in drug clinical trials and preclinical assays: a-priori ordered hypotheses. In: Vollmar J (Hrsg) Biometrie in der Chemisch-Pharmazeutischen Industrie 6. Fischer-Verlag, Stuttgart, S 3–18

  • Morikawa T, Yoshida M (1995) A useful testing strategy in phase 3 trials: combined test of superiority and test of equivalence. J Biopharm Stat 5:297–306

  • Pigeot I (2000) Basic concepts of multiple tests: a survey. Stat Pap 41:3–36

  • Rao CV, Swarupchand U (2009) Multiple comparison procedures: a note and a bibliography. J Stat 16:66–109

  • Rom DM (1990) A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika 77:663–665

  • Roy SN (1953) On a heuristic method of test construction and its use in multivariate analysis. Ann Math Stat 24:220–238

  • Ryan TP, Woodwall WH (2005) The most-cited statistical papers. J Appl Stat 32:461–474

  • Scheffé H (1953) A method for judging all contrasts in the analysis of variance. Biometrika 40:87–104

  • Shaffer JP (1986) Modified sequentially rejective multiple test procedures. J Am Stat Assoc 81:826–831

  • Shaffer JP (1995) Multiple hypothesis testing. Annu Rev Psychol 46:561–584

  • Šidák Z (1967) Rectangular confidence regions for the mean of multivariate normal distributions. J Am Stat Assoc 62:626–633

  • Sklar A (1996) Random variables, distribution functions, and copula: a personal look backward and forward. IMS Lect Notes Monogr Ser 28:1–4

  • Sonnemann E (1982) Allgemeine Lösungen multipler Testprobleme. EDV Med Biol 13:120–128

  • Spjøtvoll E (1972) On the optimality of some multiple comparison procedures. Ann Math Stat 43:398–411

  • Spjøtvoll E, Stoline MR (1973) An extension of the T-method of multiple comparisons to include the cases with unequal sample sizes. J Am Stat Assoc 68:975–978

  • Steel RGD (1959) A multiple comparison rank sum test: treatments versus control. Biometrics 15:560–572

  • Steel RGD (1960) A rank sum test for comparing all pairs of treatments. Technometrics 2:197–207

  • Storey JD (2003) The positive false discovery rate: a Bayesian interpretation and the q-value. Ann Stat 31:2013–2035

  • Storey JD, Taylor JE, Siegmund D (2004) Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. J R Stat Soc B 66:187–205

  • Tamhane AC, Liu L (2008) On weighted Hochberg procedures. Biometrika 95:279–294

  • Tamhane AC, Liu W, Dunnet CW (1998) A generalized step-up-down multiple test procedure. Can J Stat 26:353–363

  • Troendle JF (1996) A permutational step-up method of testing multiple outcomes. Biometrics 52:846–859

  • Tukey JW (1953) The problem of multiple comparisons. Unveröffentlichtes Manuskript. Princeton University, Princeton

  • Waller RA, Duncan DB (1969) A Bayes rule for the symmetric multiple comparisons problem. J Am Stat Assoc 64:1484–1503

  • Welsch RE (1977) Stepwise multiple comparison procedures. J Am Stat Assoc 72:566–575

  • Westfall P, Young SS (1993) Resampling-Based Multiple Testing. John Wiley & Sons, New York

  • Westfall PH, Tobias RD, Wolfinger RD (2011) Multiple Comparisons and Multiple Tests U sing the SAS System, 2. Aufl. SAS Press, Cary

  • Wiens BL (2003) A fixed-sequence Bonferroni procedure for testing multiple endpoints. Pharm Stat 2:211–215

  • Yekutieli D, Benjamini Y (1999) Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J Stat Plan Inference 82:171–196

Download references

Author information

Authors and Affiliations

  1. Quantitative Methoden in den Wirtschaftswissenschaften, Universität Augsburg, Universitätsstraße 16, 86159, Augsburg, Deutschland

    Christina C. Bartenschlager & Michael Krapp

Authors
  1. Christina C. Bartenschlager
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Krapp
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Krapp.

Additional information

Wir danken einem anonymen Gutachter für zahlreiche wertvolle Hinweise.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bartenschlager, C., Krapp, M. Theorie und Methoden multipler statistischer Vergleiche. AStA Wirtsch Sozialstat Arch 9, 107–129 (2015). https://doi.org/10.1007/s11943-015-0166-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11943-015-0166-9

Schlüsselwörter

Keywords

Search

Navigation