Which statistical test should I use?

Question

Answer

Choosing the correct statistical test can be difficult, as there are many different tests, each suitable to a specific type of data or experiment.

A common form of experiment is to assess whether two sets of measures are significantly different. Examples include determining whether a treatment is effective, or whether two populations vary significantly in a given characteristic. Usually a Student’s T-test will be suitable, but first a normality test must be performed, such as Kolmogorov-Smirnov. Student’s T-test assumes that the data are normally distributed. If this is not the case, a non-parametric test should be used instead, such as the Mann-Whitney U test.

Paired difference tests should be used if the two groups contain the same individuals or locations under different conditions or times. These have greater statistical power as some possible confounding effects are eliminated. For normally distributed data, paired t-tests or paired Z-tests should be suitable. For non-normal data, Wilcoxon’s signed-rank test or Welch’s t-test should prove more reliable and robust.

Whether these tests should be one-tailed or two-tailed depends on the nature of the hypothesis being tested. If the hypothesis is that one measure may be higher than the other, a one-tailed test should be used, as the measure being lower or the same would both count as disproving the hypothesis. If, however, the hypothesis is that the two measures are simply different, a two-tailed test should be used, as being higher or lower would both be of interest.

For more complex experiments, with multiple variables, Analysis of Variance, or ANOVA is often used to determine which, if any, of the variables produces a significant effect. This is in some ways similar to a series of t-tests. Although ANOVA can tolerate non-normal data quite well, it does assume a certain sphericity of the data (meaning that the variance of each sub-group is similar). This can be tested by Mauchly’s sphericity test. If either is violated, a Kruskal-Wallis test can be used instead.