Multiple Comparisons Testing from a Bayesian Perspective

A General Multiple Comparisons Problem

In this note, we examine a general multiple comparisons testing problem from a Bayesian viewpoint. Suppose we observe independent random samples from I normally distributed populations with equal variances. The goal of our problem is to determine which pairs of groups have equal means.

Write

$$\left \{{X}_{ij}\right \}\vert \left \{{\mu }_{i}\right \},{\sigma }^{2} \sim \text{ indep }N\left ({\mu }_{ i},{\sigma }^{2}\right ).$$

(1)

We are interested in testing H ^{a, b} : μ _a = μ _b for each a, b; a total of $I(I - 1)/2$ distinct, but related hypotheses. A typical frequentist test is based on the decision rule of accept H ^{a, b} when

$$\left \vert {\overline{X}}_{b} -{\overline{X}}_{a}\right \vert \leq {Q}_{a,b}.$$

(2)

The overall error rate is the probability of falsely rejecting any of the true hypotheses in the set H ^{a, b}. The determination of Q _{a, b} in (2) depends on how the overall error rate is to be controlled. A classical book featuring this...