Skip to main content
Springer Nature Link
Log in

Dependence aliasing and the control of family-wise error rate in multiple hypothesis testing

  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

The validation of the results obtained by hypothesis testing is of special interest in applications that deal with high-dimensional sets of variables. The use of equivocated statistical methods may result in poor control of false positives. On the other hand, overconservative methods may prevent relevant findings. In this paper we define dependence aliasing as the spurious dependence relationship among variables that appears when the number of samples is lesser than the number of variables of a study. We present a novel method for estimating the adjusted p-values in applications that require multiple hypothesis testing. The method increases the statistical power of the results by exploring the dependence among the variables, while controlling false positives in strong sense. The method is compared to other relevant adjustment models such as the false discovery rate method and resampling. We illustrate the effectiveness of the method in medical imaging studies involving progressively larger sets of variables. The results show that the proposed method is able to compute adjusted p-values that are closer to the ones obtained by resampling, but at a much lower computational cost.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Avants, B., Gee, J.C.: Soft parametric curve matching in scale-space. In: Sonka, M., Fitzpatrick, J.M. (eds.) Proceedings of the SPIE Medical Imaging 2002: Image Processing, pp. 1139–1151. Bellingham, San Diego (2002)

    Chapter  Google Scholar 

  • Bender, R., Lange, S.: Adjusting for multiple testing: when and how? J. Clin. Epidemiol. 54, 343–349 (2001)

    Article  Google Scholar 

  • Benjamini, B., Yekutieli, D.: The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 29(4), 1165–1188 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  • Benjamini, Y., Hochberg, T.: Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. B 85, 289–300 (1995)

    MathSciNet  Google Scholar 

  • Black, M.A.: A note on the adaptive control of false discovery rates. J. R. Stat. Soc. B 66(2), 297–304 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  • Bonferroni, C.E.: Teoria statistica delle classi e calcolo delle probabilità. R Istituto Superiore di Scienze Economiche e Commerciali di Firenze 8, 3–62 (1936)

    Google Scholar 

  • Brombin, C., Salmaso, L.: Multi-aspect permutation tests in shape analysis with small sample size. Comput. Stat. Data Anal. 53(12), 3921–3931 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  • Buchheister, B., Lehmacher, W.: Multiple testing procedures for identifying desirable dose combinations in bifactorial designs. GMS Medizinische Informatik, Biometrie und Epidemiologie 2(2), 1–11 (2006)

    Google Scholar 

  • Cheverud, J.M.: A simple correction for multiple comparisons in interval mapping genome scans. Heredity 87, 52–58 (2001)

    Article  Google Scholar 

  • Chung, F.R.K.: Spectral graph theory. American Mathematical Society, Fresno (1997)

    MATH  Google Scholar 

  • Dudoit, S., Shaffer, J.C., Boldrick, J.P.: Multiple hypothesis testing in microarray experiments. Stat. Sci. 18(1), 71–103 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  • Evans, A., Collins, D., Brown, E., Kelly, R., Petters, T.: A 3D statistical neuroanatomical models for 305 MRI volumes. In: IEEE Nuclear Science Symposium/Medical Imaging Conference, pp. 1813–1817 (1993)

  • Finos, L., Salmaso, L.: A new nonparametric approach for multiplicity control: optimal subset procedures. Comput. Stat. 20(4), 643–654 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  • Friston, K., Ashburner, J., Kiebel, S., Nichols, T., Penny, W. (eds.): Statistical parametric mapping: the analysis of functional brain images. Academic Press, London (2007)

  • Golland, P., Fischl, B.: Permutation tests for classification: towards statistical significance in image-based studies. Lecture Notes in Computer Science 2732, 330–341 (2003)

  • Grossman, M., McMillan, C., Moore, P., Ding, L., Glosser, G., Work, M., Gee, J.: What’s in a name: voxel-based morphometric analyses of MRI and naming difficulty in alzheimer’s disease, frontotemporal dementia and corticobasal degeneration. Brain 127, 1–22 (2004)

    Article  Google Scholar 

  • Gur, R.E., Turetsky, B.I., Bilker, W.D., Gur, R.C.: Reduced gray matter volume in schizophrenia. Arch. General Psychiatr. 56, 905–911 (1999)

    Article  Google Scholar 

  • Hochberg, Y.: A sharper bonferroni procedure for multiple tests of significance. Biometrika 75, 800–803 (1988)

    Article  MATH  MathSciNet  Google Scholar 

  • Holm, S.: A simple sequential rejective multiple test procedure. Scand. J. Stat. 6, 65–70 (1979)

    MATH  MathSciNet  Google Scholar 

  • Hommel, G.: A stagewise rejective multiple test procedure on a modified bonferroni test. Biometrika 75, 383–386 (1988)

    Article  MATH  Google Scholar 

  • Hwang, Y.T., Kuo, H.C., Wang, C.C., Lee, M.: Estimating the number of true null hypotheses in multiple hypothesis testing. Stat. Comput. (2013). doi:10.1007/s11222-013-9377-5

  • Li, J., Ji, L.: Adjusting multiple testing in multilocus analyses using the eigenvalues of a correlation matrix. Heredity 95, 221–227 (2005)

    Article  Google Scholar 

  • Machado, A.: Multiple testing correction in medical image analysis. J. Math. Imaging Vis. 29(2), 107–117 (2007)

    Article  MathSciNet  Google Scholar 

  • Marcus, R., Peritz, E., Gabriel, K.R.: On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63, 655–660 (1976)

    Google Scholar 

  • Nichols, T., Hayasaka, S.: Controlling the familywise error rate in functional neuroimaging: a comparative review. Stat. Methods Med. Res. 12(5), 419–446 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  • Nichols, T., Holmes, A.: Nonparametric permutation tests for functional neuroimaging: a primer with examples. Hum. Brain Mapp. 15, 1–25 (2001)

    Article  Google Scholar 

  • Nyholt, D.R.: A simple correction for multiple testing for single-nucleotide polymorphisms in linkage disequilibrium with each other. Am. J. Hum. Genet. 74, 765–769 (2004)

    Article  Google Scholar 

  • Pappas, T.N.: An adaptive clustering algorithm for image segmentation. IEEE Trans. Signal Process. 40, 901–914 (1992)

    Article  Google Scholar 

  • Perneger, T.V.: What’s wrong with bonferroni adjustments. BMJ 316, 1236–1238 (1998)

    Article  Google Scholar 

  • Pesarin, F., Salmaso, L.: Permutation tests for complex data: theory, applications and software. Wiley, New York (2010)

    Google Scholar 

  • Pesarin, F., Salmaso, L.: A review and some new results on permutation testing for multivariate problems. Stat. Comput. 22(2), 639–646 (2012)

    Article  MathSciNet  Google Scholar 

  • Romano, J.R., Wolf, M.: Exact and approximate stepdown methods for multiple hypothesis testing. J. Am. Stat. Assoc. 100(469), 94–108 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  • Rosenfeld, A., Kak, A.: Digital picture processing. Academic Press, Orlando (1982)

    Google Scholar 

  • Rothman, K.J.: No adjustments are needed for multiple comparisons. Epidemiology 1, 43–46 (1990)

    Article  Google Scholar 

  • Salyakina, D., Seaman, S.R., Browning, B.L., Dudbridge, F., Muller-Myhsok, B.: Evaluation of nyholt’s procedure for multiple testing correction. Hum. Hered. 60, 19–25 (2005)

    Article  Google Scholar 

  • Shtasel, D.L., Gur, R.E., Mozley, P.D., Richards, J., Taleff, M.M., Heimberg, C., Gallacher, F., Gur, R.C.: Voluntiers for biomedical research: recruitment and screening of normal controls. Arch. General Psychiatr. 48, 1022–1025 (1991)

    Article  Google Scholar 

  • Sidak, Z.: Rectangular confidence regions for the means of multivariate normal distributions. J. Am. Stat. Assoc. 62, 626–633 (1967)

    MATH  MathSciNet  Google Scholar 

  • Simes, J.R.: An improved bonferroni procedure for multiple test of significance. Biometrika 73(3), 751–754 (1986)

    Article  MATH  MathSciNet  Google Scholar 

  • Strang, G.: Linear algebra and its applications. Academic Press, San Diego (1980)

    Google Scholar 

  • Thompson, P.M., Moussai, J., Zohoori, S., Goldkorn, A., Khan, A., Mega, M.S., Small, J., Cummings, J., Toga, A.W.P.: Cortical variability and assymetry in normal aging and Alzheimer’s disease. Cereb. Cortex 8, 492–509 (1998)

    Article  Google Scholar 

  • Wagner, G.: On the eigenvalue distribution of genetic and phenotypic dispersion matrices: evidence for a nonrandom organization of quantitative character variation. J. Math. Biol. 21, 77–95 (1984)

    Article  MATH  MathSciNet  Google Scholar 

  • Westfall, P., Zaykin, D., Young, S.: Multiple tests for genetic effects in association studies. In: Looney, S. (ed.) Biostatistical methods, methods in molecular biology, vol. 184, pp. 143–168. Humana Press, Totowa (2002)

    Chapter  Google Scholar 

  • Westfall, P.H., Tobias, R.: Multiple testing of general contrasts: truncated closure and the extended shaffer-royan method. J. Am. Stat. Assoc. 102(478), 487–494 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  • Westfall, P.H., Troendle, J.F.: Multiple testing with minimal assumptions. Biom. J. 50(5), 745–755 (2008)

    Article  MathSciNet  Google Scholar 

  • Westfall, P.H., Young, S.S.: Resampling-based multiple testing: examples and methods for p-value adjustment. Wiley, New York (1993)

    Google Scholar 

Download references

Acknowledgments

The author is grateful to the University of Pennsylvania for sharing the callosum and AD data. This work was partly supported by CNPq-Brazil (481989/2010-2, 301907/2010-2), FAPEMIG (PPM 00416/11) and INCT-MM (FAPEMIG: CBB-APQ-00075-09 / CNPq 573646/2008-2)

Author information

Authors and Affiliations

  1. Department of Computer Science and Graduate Program in Electrical Engineering, Pontifical Catholic University of Minas Gerais, Belo Horizonte, Brazil

    Alexei Manso Correa Machado

  2. Department of Anatomy and imaging, Federal University of Minas, Gerais Medical School, Belo Horizonte, Brazil

    Alexei Manso Correa Machado

Authors
  1. Alexei Manso Correa Machado
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Alexei Manso Correa Machado.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Machado, A.M.C. Dependence aliasing and the control of family-wise error rate in multiple hypothesis testing. Stat Comput 25, 669–681 (2015). https://doi.org/10.1007/s11222-014-9459-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-014-9459-z

Keywords