Skip to main content
SpringerLink
Log in

Estimating the number of true null hypotheses in multiple hypothesis testing

  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

The overall Type I error computed based on the traditional means may be inflated if many hypotheses are compared simultaneously. The family-wise error rate (FWER) and false discovery rate (FDR) are some of commonly used error rates to measure Type I error under the multiple hypothesis setting. Many controlling FWER and FDR procedures have been proposed and have the ability to control the desired FWER/FDR under certain scenarios. Nevertheless, these controlling procedures become too conservative when only some hypotheses are from the null. Benjamini and Hochberg (J. Educ. Behav. Stat. 25:60–83, 2000) proposed an adaptive FDR-controlling procedure that adapts the information of the number of true null hypotheses (m 0) to overcome this problem. Since m 0 is unknown, estimators of m 0 are needed. Benjamini and Hochberg (J. Educ. Behav. Stat. 25:60–83, 2000) suggested a graphical approach to construct an estimator of m 0, which is shown to overestimate m 0 (see Hwang in J. Stat. Comput. Simul. 81:207–220, 2011). Following a similar construction, this paper proposes new estimators of m 0. Monte Carlo simulations are used to evaluate accuracy and precision of new estimators and the feasibility of these new adaptive procedures is evaluated under various simulation settings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Benjamini, Y., Hochberg, Y.: Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc., Ser. B 57, 289–300 (1995)

    MATH  MathSciNet  Google Scholar 

  • Benjamini, Y., Hochberg, Y.: On the adaptive control of the false discovery rate in multiple testing with independent statistics. J. Educ. Behav. Stat. 25, 60–83 (2000)

    Article  Google Scholar 

  • Benjamini, Y., Liu, W.: A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. J. Stat. Plan. Inference 82, 163–170 (1999a)

    Article  MATH  MathSciNet  Google Scholar 

  • Benjamini, Y., Liu, W.: A distribution-free multiple test procedure that controls the false discovery rate. Unpublished manuscript (1999b)

  • Benjamini, Y., Yekutieli, D.: The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 29, 1165–1188 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  • Benjamini, Y., Krieger, A.M., Yekutieli, A.: Adaptive linear step-up procedures that control the false discovery rate. Biometrika 93(3), 491–507 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  • Gavrilov, Y., Benjamini, Y., Sarkar, S.K.: An adaptive step-down procedure with proven FDR control under independence. Ann. Stat. 37, 619–629 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  • Ge, Y., Dudoit, S., Speed, T.P.: Resampling-based multiple testing for microarray data hypothesis. Test 12, 1–44 (2003)

    Article  MathSciNet  Google Scholar 

  • Genovese, C., Wasserman, L.: A stochastic process approach to false discovery control. Ann. Stat. 32, 1035–1061 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  • Holm, S.: A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6, 65–70 (1979)

    MATH  MathSciNet  Google Scholar 

  • Hochberg, Y.: A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, 800–803 (1988)

    Article  MATH  MathSciNet  Google Scholar 

  • Hochberg, Y., Benjamini, Y.: More powerful procedures for multiple significance testing. Stat. Med. 9, 811–818 (1990)

    Article  Google Scholar 

  • Hsueh, H.M., Chen, J.J., Kodell, R.L.: Comparison of methods for estimating the number of true null hypotheses in multiplicity testing. J. Biopharm. Stat. 13, 675–689 (2003)

    Article  MATH  Google Scholar 

  • Hsueh, H.M., Tsai, C.A., Chen, J.J.: Incorporating the number of true null hypotheses to improve power in multiple testing: application to gene microarray data. J. Stat. Comput. Simul. 77, 757–767 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  • Hwang, Y.T.: Estimating the number of true null hypotheses in multiple hypotheses testing. J. Stat. Comput. Simul. 81, 207–220 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  • Hwang, Y.T., Lai, J.J., Ou, S.T.: Evaluations of FWER-controlling methods in multiple hypothesis testing. J. Appl. Stat. 37, 1681–1694 (2010)

    Article  MathSciNet  Google Scholar 

  • Hwang, Y.T., Chu, S.K., Ou, S.T.: Evaluations of FDR-controlling methods in multiple hypothesis testing. Stat. Comput. 21, 569–583 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  • Langaas, M., Lindqvist, B.H., Ferkingstad, E.: Estimating the proportion of true null hypotheses, with application to DNA microarray data. J. R. Stat. Soc., Ser. B 67, 555–572 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  • Liu, F., Sarkar, S.K.: A new adaptive method to control the false discovery rate. In: Bhattacharjee, M., Dhar, S.K., Subramanian, S. (eds.) Recent Advances in Biostatistics: False Discovery Rates, Survival Analysis, and Related Topics. Series in Biostatistics, vol. 4, pp. 3–26. World Scientific, New Jersey (2011)

    Chapter  Google Scholar 

  • Lu, X., Perkins, D.L.: Resampling strategy to improve the estimation of number of null hypotheses in FDR control under strong correlation structure. BMC Bioinform. 8, 157–169 (2007)

    Article  Google Scholar 

  • Meinshausen, N.: False discovery control for multiple tests of association under general dependence. Scand. J. Stat. 33(2), 227–237 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  • Meinshausen, N., Rice, J.: Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses. Ann. Stat. 34, 373–393 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  • Miller, C.J., Genovese, C., Nichol, R.C., Wasserman, L., Connolly, A., Reichart, D., Hopkins, A., Schneider, J., Moore, A.: Controlling the false-discovery rate in astrophysical data analysis. Astron. J. 122, 3492–3505 (2001)

    Article  Google Scholar 

  • Pawian, Y., Michiels, S., Koscielny, S., Gusnanto, A., Ploner, A.: False discovery rate, sensitivity and sample size for microarray studies. Bioinformatics 21, 3017–3024 (2005)

    Article  Google Scholar 

  • Pounds, S.B.: Estimation and control of multiple testing error rates for microarray studies. Brief. Bioinform. 7, 25–36 (2006)

    Article  Google Scholar 

  • R Development Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna (2008). http://www.R-project.org. ISBN 3-900051-07-0

    Google Scholar 

  • Schweder, T., Spjøtvoll, E.: Plots of p-values to evaluate many test simultaneously. Biometrika 69, 493–502 (1982)

    Google Scholar 

  • Seeger, P.: A note on a method for the analysis of significance en masse. Technometrics 10, 583–593 (1968)

    Article  Google Scholar 

  • Storey, J.D.: A direct approach to false discovery rates. J. R. Stat. Soc., Ser. B 64, 479–498 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  • Storey, J.D., Taylor, J.E., Siegmund, D.: Strong control, conservative point estimation, and simultaneous conservative consistency of false discovery rates: a unified approach. J. R. Stat. Soc., Ser. B 66, 187–205 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  • Turkheimer, F.E., Smith, C.B., Schmidt, K.: Estimation of the number of true null hypotheses in multivariate analysis of neuroimaging data. NeuroImage 13, 920–930 (2001)

    Article  Google Scholar 

  • Wang, C.C., Lin, Y.H., Hwang, Y.T.: A robust estimation of the proportion of true null hypotheses based on a beta mixture model. J. Chin. Stat. Assoc. 49, 1–17 (2011)

    Google Scholar 

  • Yekutieli, D., Benjamini, Y.: Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. J. Stat. Plan. Inference 82, 171–196 (1999)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors gratefully acknowledge the Associate Editors and the referees for their insightful comments, which enhanced greatly the presentation and methodology of this paper. This research is partially supported by Nation Science Council Grant # NSC 96-2118-M-305-001 and # NSC 99-2118-M-305-001.

Author information

Authors and Affiliations

  1. Department of Statistics, National Taipei University, Taipei, Taiwan

    Yi-Ting Hwang, Chun-Chao Wang & Meng Feng Lee

  2. Department of Statistics, National Chengchi University, Taipei, Taiwan

    Hsun-Chih Kuo

Authors
  1. Yi-Ting Hwang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hsun-Chih Kuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chun-Chao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Meng Feng Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yi-Ting Hwang.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

(PDF 481 kB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hwang, YT., Kuo, HC., Wang, CC. et al. Estimating the number of true null hypotheses in multiple hypothesis testing. Stat Comput 24, 399–416 (2014). https://doi.org/10.1007/s11222-013-9377-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-013-9377-5

Keywords

Search

Navigation