Abstract
Permutation or randomization test is a nonparametric test in which the null distribution (distribution under the null hypothesis of no relationship or no effect) of the test statistic is attained by calculating the values of the test statistic overall permutations (or by considering a large number of random permutation) of the observed dataset. The power of permutation test evaluated based on the observed dataset is called conditional power. In this paper, the conditional power of permutation tests is reviewed. The use of the conditional power function for sample size estimation is investigated. Moreover, reproducibility and generalizability probabilities are defined. The use of these probabilities for sample size adjustment is shown. Finally, an illustration example is used.


Similar content being viewed by others
References
Aguirre-Urreta M, Rönkkö M (2015) Sample size determination and statistical power analysis in PLS using R: an annotated tutorial. Commun Assoc Inf Syst 36(3):33–51
Akobeng AK (2016) Understanding type I and type II errors, statistical power and sample size. Acta Paediatr 105(6):605–609
Amro L, Pauly M (2017) Permuting incomplete paired data: a novel exact and asymptotic correct randomization test. J Stat Comput Simul 87(6):1148–1159
Barton DE (1957) A comparison of two sorts of test for a change of location applicable to truncated data. J R Stat Soc 19:119–124
Basso D, Pesarin F, Salmaso L, Solari A (2009) Permutation tests for stochastic ordering and ANOVA: theory and applications in R. Springer, New York
Bell CB, Moser JM, Thompson R (1966) Goodness criteria for two-sample distribution-free tests. Ann Math Stat 37:133–142
Brewer JK, Sindelar PT (1988) Adequate sample size: a priori and post hoc considerations. J Spec Educ 21:74–84
Chow SC, Liu JP (2004) Design and analysis of clinical trials: concepts and methodologies, 2nd edn. Wiley-Blackwell, New York
Chow SC, Shao J, Wang H (2002) A note on sample size calculation for mean comparisons based on non-central \(t\)-statistics. J Biopharm Stat 12:441–456
Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, Hillsdale
Collings BJ, Hamilton MA (1988) Estimating the power of the two-sample Wilcoxon test for location shift. Biometrics 44:847–860
Cooper H, Hedges LV (1997) The handbook of research synthesis. Russell Sage Foundation, New York
De Capitani L, De Martini D (2016) Reproducibility probability estimation and RP-testing for some nonparametric tests. Entropy 18(4):142
De Martini D (2002) Pointwise estimate of the power and sample size determination for permutation tests. Statistica 62:779–790
Dixon WJ (1954) Power under normality of several nonparametric tests. Ann Math Stat 25:610–614
Edgington ES (1995) Randomization tests, 3rd edn. Marcel Dekker, New York
Epstein B (1955) Comparison of some non-parametric tests against normal alternatives with an application to life testing. J Am Stat Assoc 50:894–900
Fisher RA (1934) Statistical methods for research workers. Oliver and Boyd, Edinburgh
Fisher RA (1935) The design of experiments. Oliver and Boyd, Edinburgh
Giraudeau B, Higgins J, Tavernier E, Trinquart L (2016) Sample size calculation for meta-epidemiological studies. Stat Med 35(2):239–250
Good P (2005) Permutation, parametric and bootstrap tests of hypotheses, 3rd edn. Springer, New York
Goodman S (1992) A comment on replication, \(p\)-values and evidence. Stat Med 11:875–879
Hallahan M, Rosenthal R (1996) Statistical power: concepts, procedures, and applications. Behav Res Ther 34:489–499
Hamilton MA, Collings BJ (1991) Determining the appropriate sample size for nonparametric tests for location shift. Technometrics 33:327–337
Haynam GE, Govindarajulu Z (1966) Exact power of the Mann–Whitney test for exponential and rectangular alternatives. Ann Math Stat 37:945–953
Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic Press, New York
Hemelrijk J (1961) Experimental comparison of Student’s and Wilcoxon’s two sample test. Quantitative Methods in Pharmacology. Inter-science, New York, pp 118–133
Hoeffding W (1952) The large-sample power of tests based on permutations of observations. Ann Math Stat 23:169–192
Kraemer HC, Thiemann S (1987) How many subjects? Statistical power analysis in research. Sage Publications, Newbury Park
Lehmann EL, Romano JP (2005) Testing statistical hypotheses, 3rd edn. Springer, New York
Lehmann EL, Stein C (1949) On the theory of some non-parametric hypotheses. Ann Math Stat 20:28–45
Lenth RV (2007) Post hoc power: tables and commentary. Technical Report 378, The University of Iowa - Department of Statistics and Actuarial Science
Levine M, Ensom MHH (2001) Post hoc power analysis: an idea whose time has passed? Pharmacotherapy 21:405–409
Lipsey MW (1990) Design sensitivity: statistical power for experimental research. Sage Publications, Newbury Park
Markowski EP, Markowski CA (1999) Practical uses of statistical power in business research studies. J Educ Bus 75:122–125
McDonald J, Gerard PD, McMahan CS, Schucany WR (2016) Exact-permutation-based sign tests for clustered binary data via weighted and unweighted test statistics. J Agric Biol Environ Stat 21(4):698–712
McHugh RB (1961) Confidence interval inference and sample size determination. Am Stat 15:14–17
Milton RC (1970) Rank order probabilities: two-sample normal shift alternatives. Wiley, New York
Moher D, Dulberg CS, Wells GA (1994) Statistical power, sample size, and their reporting in randomized controlled trials. J Am Med Assoc 272:122–124
Noether GE (1987) Sample size determination for some common nonparametric tests. J Am Stat Assoc 82:645–647
Onwuegbuzie AJ, Leech NL (2004) Post hoc power: a concept whose time has come. Underst Stat 3:201–230
Owen DB (1965) The power of Student’s \(t\)-test. J Am Stat Assoc 60:320–333
Pesarin F (2001) Multivariate permutation tests: with application in biostatistics. Wiley, Chichester
Pesarin F, Salmaso L (2010) Permutation tests for complex data: theory application and software. Wiley, Chichester
Pitman EJG (1937a) Significance tests which may be applied to samples from any population. J R Stat Soc B 4:119–130
Pitman EJG (1937b) Significance tests which may be applied to samples from any population. II. The correlation coefficient test. J R Stat Soc B 4:225–232
Pitman EJG (1938) Significance tests which may be applied to samples from any population. III. The analysis of variance test. Biometrika 29:322–335
Randles RH, Wolfe DA (1979) Introduction to the theory of nonparametric statistics. Wiley, New York
Salmaso L (2003) Synchronized permutation tests in \(2^k\) factorial designs. Commun Stat Theory Methods 32:1419–1437
Samonenko I, Robinson J (2015) A new permutation test statistic for complete block designs. Ann Stat 43(1):90–101
Schmitt MC (1987) The effects on an elaborated directed reading activity on the metacomprehension skills of third graders. Ph.D. Thesis, Purdue University
Shao J, Chow SC (2002) Reproducibility probability in clinical trials. Stat Med 21:1727–1742
Shorack GR, Wellner JA (1986) Empirical processes with applications to statistics. Wiley Series in Probability & Mathematical Statistics, New York
Simonoff JS, Hochberg Y, Reiser B (1986) Alternative estimation procedures for \({P}_r({X} < {Y})\) in categorized data. Biometrics 42:895–907
Teichroew D (1955) Empirical power functions for nonparametric two-sample tests for small samples. Ann Math Stat 26:340–344
Thomas L (1997) Retrospective power analysis. Conserv Biol 11:276–280
Wang H, Chow SC, Chen M (2005) A Bayesian approach on sample size calculation for comparing means. J Biopharm Stat 15:799–807
Acknowledgements
The authors would like to thank the associated editor and referees for their comments that contribute in improving the paper. We also greatly appreciate Dr. Ibrahim Almasri, Department of Applied Mathematics and Physics - Palestine Polytechnic University, for being kind enough to read and improve the language of the paper.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Samuh, M.H., Pesarin, F. Applications of conditional power function of two-sample permutation test. Comput Stat 33, 1847–1862 (2018). https://doi.org/10.1007/s00180-018-0803-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00180-018-0803-1