Interpreting Effect Size Estimates through Graphic Analysis of Raw Data Distributions

Abstract

Effect size estimates are altered by many factors, including, and perhaps most importantly, the shapes of compared distributions. There have been many long time advocates of the necessity of graphing raw data to truly understand analysis.  Though they were and remain correct, there is little evidence in the published literature in psychology that their recommendations have been followed.  This paper argues their case, but with the advantage of the recent emphasis on effect sizes promoted by, amongst others, the American Psychological Association publication guide.  Unlike Null Hypothesis Statistical Testing (NHST), effect size estimates are not robust to distributional deviations from normality.  As a consequence of effect size sensitivity to distributional distortions from normality, it is all the more important to understand the qualities of the distributions from which estimates are derived.  In this paper, we consider and simulate cases where graphical analyses reveal distortion in effect size estimates, and in doing so highlight the value of graphing data to interpret effect size estimates.