Title: Conclusion
Course: UGQuants (SSPS) > Section: Inference

Conclusion

In this module, we have seen how to apply the logic of sampling inference to a very practical problem. That is, how can we infer the existence of relationships between variables in a population through results obtained in a random sample of that population. Further we have learned to use SPSS to produce the appropriate test statistics, and calculate confidence intervals and associated levels of statistical significance (or the probability of committing a type I error).

This procedure allows us to use Chi Square to test for relationships between variables in the same way that we could use a confidence interval for interval level variables. The Chi Square test, as performed by SPSS, gives us a direct measure of the chances of committing a type I error if we reject a null hypothesis of no association between two variables. Thus if the p value, or level of significance, associated with the value for Chi Sq in a table is low enough (e.g. conventionally below 0.05 or 0.01) we can provisionally accept that we have evidence of a relationship not only in our sample, but also in the population from which our sample was drawn.

However, a positive Chi Sq, or any other 'statistically significant' result only tells us about the relationship between our sample and our population. It does NOT tell us if something is 'significant' in the sense of important. That comes from our theory and the substance of our empirical findings.

If we fail to keep this distinction in mind then we risk falling into two common errors.  We can avoid these is we remember two important points

1
The first is that correlation or association are quite distinct from significance: do not confuse them!
Association or correlation refers to the strength of a relationship in our data, whether the cases in our data constitute a random sample, or an entire population or some other group.
Significance refers exclusively to the relationship between a random sample and a population.
The confusion between these measures arises because the stronger the association between a pair of variables in data from a random sample, the more likely it is that such a relationship will also be significant. However, the vital missing element here is sample size. With a small enough sample we can often be unsure if even the strongest correlations are significant.
2 | The second error to avoid is that of 'fishing' in random sample data for apparently 'significant' results. If you use a 5% level of significance as your test, on average one in twenty (5%) of the results you find will indeed be due to chance! The moral of this story is: use theory to drive your exploration of relationships between variables, not the calculating power of your computer!

It is important, paradoxically, not to get too obsessed with significance tests!  This idea may make you a little frustrated (why spend so much effort learning all this if the next thing we discover is that we shouldn't pay too much attention to it …) but in fact it is another case of statistics being about informed judgments rather than blind computation or calculation.

If relationships are not strong enough for us to decide the risk of a type I error is small enough to disregard - in other words, if we decide it is best to accept the null hypothesis - then usually we need go no further. In some circumstances, though, we might want to think about whether we have set too stringent a level of error, especially if we are exploring our data to see what might be there, rather than looking for evidence to accept or refute a particular idea or argument.

Conversely, one of the least useful things we can do is to decide that everything that pops up as statistically significant in our data must be important or interesting.

Why? Well, with a little reflection we can see that there are two dangers lurking here. First, if we have a large enough sample size, even the tiniest, most irrelevant relationship will nevertheless appear significant because we have enough information to capture even fine details. Thus, when we encounter a relationship that is significant, we nevertheless still have to ask ourselves, is it substantial. The meaning of substantial will also depend on what we are interested in.

The lesson is the usual one: rather than just routinely applying the rules, think about what you are doing! As Cathie Marsh put it, statistics is for detectives, not lawyers!

In working through this module you should have learned...

1 | The procedure for establishing a null hypothesis for a population.
2 | The decision to set an acceptable level of error for testing the hypothesis using random sample data.
3 | How to calculate Chi Square for two categorical variables.
4 | How to use statistical software to do this and interpret its results.
5 | How to decide whether to accept or reject an appropriately formulated null hypothesis, given the level of risk of committing a type I error.
6 | The importance of significance testing and its limits.