Rationale: Approaching statistical significance means that a result did not meet the predetermined level of significance, which is usually .05, but was close. In some ways it is a “nicer way of admitting that your results support the null hypothesis”.¹ While it is tempting to report a result as approaching significance, it is problematic. A decision was made regarding the acceptable standard and now, based on the results, that decision is being modified to fit the data. This is a violation of the assumptions behind inferential statistics. The definition of “close” is problematic since it is based on what was found rather than any scientific rationale. If results are approaching significance it might be possible to collect data from more participants until the results are significant or until the significance gets worse. Another option would be to replicate the study.²

Rationale: Since statistical significance testing is based on a probability, the more statistical tests you do, the more likely you are to get statistically significant results that are incorrect.³ This cumulative error rate means that it is inappropriate to do 100 tests and then report as significant the five comparisons reaching the .05 level because of the dangers of capitalizing on chance. The major exception would be if the tests were intended to be exploratory and would be used only in a subsequent independent study to help generate hypotheses.

Rationale: A finding of statistical significance means that the null hypotheses -- the hypotheses of no difference -- has been rejected. By itself, it does not mean that the findings are important or meaningful. It is important to determine the size or magnitude of the effects found. Effect sizes can be computed for group differences and for correlations.⁵ One well known effect size is Cohen's d. Cohen defined effect sizes as “small, d = .2,” “medium, d = .5,” and “large, d = .8,” but warned of the risks inherent in defining the terms in as “diverse a field of inquiry as behavioral science”.⁶ Larger effect sizes indicate that not only is a difference statistically significant but the likelihood is that the difference is meaningful as well.

Rationale: Data can be analyzed in multiple ways, each of which could yield legitimate answers. There are a number of factors that determine whether an analysis is appropriate including the number of dependent (or outcome) variables; the nature of the independent (or predictor) variables; whether your dependent variable is an interval variable, ordinal, or categorical variable⁸; and whether it is normally distributed.⁹ Use of an inappropriate statistic can lead to inaccurate results.