Skip to main content
Log in

Applications of conditional power function of two-sample permutation test

  • Original Paper
  • Published:
Computational Statistics Aims and scope Submit manuscript

Abstract

Permutation or randomization test is a nonparametric test in which the null distribution (distribution under the null hypothesis of no relationship or no effect) of the test statistic is attained by calculating the values of the test statistic overall permutations (or by considering a large number of random permutation) of the observed dataset. The power of permutation test evaluated based on the observed dataset is called conditional power. In this paper, the conditional power of permutation tests is reviewed. The use of the conditional power function for sample size estimation is investigated. Moreover, reproducibility and generalizability probabilities are defined. The use of these probabilities for sample size adjustment is shown. Finally, an illustration example is used.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Aguirre-Urreta M, Rönkkö M (2015) Sample size determination and statistical power analysis in PLS using R: an annotated tutorial. Commun Assoc Inf Syst 36(3):33–51

    Google Scholar 

  • Akobeng AK (2016) Understanding type I and type II errors, statistical power and sample size. Acta Paediatr 105(6):605–609

    Article  Google Scholar 

  • Amro L, Pauly M (2017) Permuting incomplete paired data: a novel exact and asymptotic correct randomization test. J Stat Comput Simul 87(6):1148–1159

    Article  MathSciNet  Google Scholar 

  • Barton DE (1957) A comparison of two sorts of test for a change of location applicable to truncated data. J R Stat Soc 19:119–124

    MathSciNet  MATH  Google Scholar 

  • Basso D, Pesarin F, Salmaso L, Solari A (2009) Permutation tests for stochastic ordering and ANOVA: theory and applications in R. Springer, New York

    MATH  Google Scholar 

  • Bell CB, Moser JM, Thompson R (1966) Goodness criteria for two-sample distribution-free tests. Ann Math Stat 37:133–142

    Article  MathSciNet  MATH  Google Scholar 

  • Brewer JK, Sindelar PT (1988) Adequate sample size: a priori and post hoc considerations. J Spec Educ 21:74–84

    Article  Google Scholar 

  • Chow SC, Liu JP (2004) Design and analysis of clinical trials: concepts and methodologies, 2nd edn. Wiley-Blackwell, New York

    MATH  Google Scholar 

  • Chow SC, Shao J, Wang H (2002) A note on sample size calculation for mean comparisons based on non-central \(t\)-statistics. J Biopharm Stat 12:441–456

    Article  Google Scholar 

  • Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, Hillsdale

    MATH  Google Scholar 

  • Collings BJ, Hamilton MA (1988) Estimating the power of the two-sample Wilcoxon test for location shift. Biometrics 44:847–860

    Article  MATH  Google Scholar 

  • Cooper H, Hedges LV (1997) The handbook of research synthesis. Russell Sage Foundation, New York

    Google Scholar 

  • De Capitani L, De Martini D (2016) Reproducibility probability estimation and RP-testing for some nonparametric tests. Entropy 18(4):142

    Article  Google Scholar 

  • De Martini D (2002) Pointwise estimate of the power and sample size determination for permutation tests. Statistica 62:779–790

    MathSciNet  MATH  Google Scholar 

  • Dixon WJ (1954) Power under normality of several nonparametric tests. Ann Math Stat 25:610–614

    Article  MathSciNet  MATH  Google Scholar 

  • Edgington ES (1995) Randomization tests, 3rd edn. Marcel Dekker, New York

    MATH  Google Scholar 

  • Epstein B (1955) Comparison of some non-parametric tests against normal alternatives with an application to life testing. J Am Stat Assoc 50:894–900

    MathSciNet  MATH  Google Scholar 

  • Fisher RA (1934) Statistical methods for research workers. Oliver and Boyd, Edinburgh

    MATH  Google Scholar 

  • Fisher RA (1935) The design of experiments. Oliver and Boyd, Edinburgh

    Google Scholar 

  • Giraudeau B, Higgins J, Tavernier E, Trinquart L (2016) Sample size calculation for meta-epidemiological studies. Stat Med 35(2):239–250

    Article  MathSciNet  Google Scholar 

  • Good P (2005) Permutation, parametric and bootstrap tests of hypotheses, 3rd edn. Springer, New York

    MATH  Google Scholar 

  • Goodman S (1992) A comment on replication, \(p\)-values and evidence. Stat Med 11:875–879

    Article  Google Scholar 

  • Hallahan M, Rosenthal R (1996) Statistical power: concepts, procedures, and applications. Behav Res Ther 34:489–499

    Article  Google Scholar 

  • Hamilton MA, Collings BJ (1991) Determining the appropriate sample size for nonparametric tests for location shift. Technometrics 33:327–337

    Article  MathSciNet  MATH  Google Scholar 

  • Haynam GE, Govindarajulu Z (1966) Exact power of the Mann–Whitney test for exponential and rectangular alternatives. Ann Math Stat 37:945–953

    Article  MathSciNet  MATH  Google Scholar 

  • Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic Press, New York

    MATH  Google Scholar 

  • Hemelrijk J (1961) Experimental comparison of Student’s and Wilcoxon’s two sample test. Quantitative Methods in Pharmacology. Inter-science, New York, pp 118–133

    Google Scholar 

  • Hoeffding W (1952) The large-sample power of tests based on permutations of observations. Ann Math Stat 23:169–192

    Article  MathSciNet  MATH  Google Scholar 

  • Kraemer HC, Thiemann S (1987) How many subjects? Statistical power analysis in research. Sage Publications, Newbury Park

    Google Scholar 

  • Lehmann EL, Romano JP (2005) Testing statistical hypotheses, 3rd edn. Springer, New York

    MATH  Google Scholar 

  • Lehmann EL, Stein C (1949) On the theory of some non-parametric hypotheses. Ann Math Stat 20:28–45

    Article  MATH  Google Scholar 

  • Lenth RV (2007) Post hoc power: tables and commentary. Technical Report 378, The University of Iowa - Department of Statistics and Actuarial Science

  • Levine M, Ensom MHH (2001) Post hoc power analysis: an idea whose time has passed? Pharmacotherapy 21:405–409

    Article  Google Scholar 

  • Lipsey MW (1990) Design sensitivity: statistical power for experimental research. Sage Publications, Newbury Park

    Google Scholar 

  • Markowski EP, Markowski CA (1999) Practical uses of statistical power in business research studies. J Educ Bus 75:122–125

    Article  MATH  Google Scholar 

  • McDonald J, Gerard PD, McMahan CS, Schucany WR (2016) Exact-permutation-based sign tests for clustered binary data via weighted and unweighted test statistics. J Agric Biol Environ Stat 21(4):698–712

    Article  MathSciNet  MATH  Google Scholar 

  • McHugh RB (1961) Confidence interval inference and sample size determination. Am Stat 15:14–17

    Google Scholar 

  • Milton RC (1970) Rank order probabilities: two-sample normal shift alternatives. Wiley, New York

    MATH  Google Scholar 

  • Moher D, Dulberg CS, Wells GA (1994) Statistical power, sample size, and their reporting in randomized controlled trials. J Am Med Assoc 272:122–124

    Article  Google Scholar 

  • Noether GE (1987) Sample size determination for some common nonparametric tests. J Am Stat Assoc 82:645–647

    Article  MathSciNet  MATH  Google Scholar 

  • Onwuegbuzie AJ, Leech NL (2004) Post hoc power: a concept whose time has come. Underst Stat 3:201–230

    Article  Google Scholar 

  • Owen DB (1965) The power of Student’s \(t\)-test. J Am Stat Assoc 60:320–333

    MathSciNet  MATH  Google Scholar 

  • Pesarin F (2001) Multivariate permutation tests: with application in biostatistics. Wiley, Chichester

    MATH  Google Scholar 

  • Pesarin F, Salmaso L (2010) Permutation tests for complex data: theory application and software. Wiley, Chichester

    Book  MATH  Google Scholar 

  • Pitman EJG (1937a) Significance tests which may be applied to samples from any population. J R Stat Soc B 4:119–130

    MATH  Google Scholar 

  • Pitman EJG (1937b) Significance tests which may be applied to samples from any population. II. The correlation coefficient test. J R Stat Soc B 4:225–232

    MATH  Google Scholar 

  • Pitman EJG (1938) Significance tests which may be applied to samples from any population. III. The analysis of variance test. Biometrika 29:322–335

    MATH  Google Scholar 

  • Randles RH, Wolfe DA (1979) Introduction to the theory of nonparametric statistics. Wiley, New York

    MATH  Google Scholar 

  • Salmaso L (2003) Synchronized permutation tests in \(2^k\) factorial designs. Commun Stat Theory Methods 32:1419–1437

    Article  MathSciNet  MATH  Google Scholar 

  • Samonenko I, Robinson J (2015) A new permutation test statistic for complete block designs. Ann Stat 43(1):90–101

    Article  MathSciNet  MATH  Google Scholar 

  • Schmitt MC (1987) The effects on an elaborated directed reading activity on the metacomprehension skills of third graders. Ph.D. Thesis, Purdue University

  • Shao J, Chow SC (2002) Reproducibility probability in clinical trials. Stat Med 21:1727–1742

    Article  Google Scholar 

  • Shorack GR, Wellner JA (1986) Empirical processes with applications to statistics. Wiley Series in Probability & Mathematical Statistics, New York

    MATH  Google Scholar 

  • Simonoff JS, Hochberg Y, Reiser B (1986) Alternative estimation procedures for \({P}_r({X} < {Y})\) in categorized data. Biometrics 42:895–907

    Article  MathSciNet  MATH  Google Scholar 

  • Teichroew D (1955) Empirical power functions for nonparametric two-sample tests for small samples. Ann Math Stat 26:340–344

    Article  MathSciNet  MATH  Google Scholar 

  • Thomas L (1997) Retrospective power analysis. Conserv Biol 11:276–280

    Article  Google Scholar 

  • Wang H, Chow SC, Chen M (2005) A Bayesian approach on sample size calculation for comparing means. J Biopharm Stat 15:799–807

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the associated editor and referees for their comments that contribute in improving the paper. We also greatly appreciate Dr. Ibrahim Almasri, Department of Applied Mathematics and Physics - Palestine Polytechnic University, for being kind enough to read and improve the language of the paper.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Monjed H. Samuh.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Samuh, M.H., Pesarin, F. Applications of conditional power function of two-sample permutation test. Comput Stat 33, 1847–1862 (2018). https://doi.org/10.1007/s00180-018-0803-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00180-018-0803-1

Keywords

Navigation