Title: Chi Sq. and statistical software
Course: UGQuants (SSPS) > Section: Inference

Chi Sq. and statistical software

Statistical software, like SPSS, Stata or R, does all these calculations for you and this, like most things, has its good side and its bad side.

The good side is that if you know what you are doing, this makes life quicker and easier. The statistical software does not only calculate the value of Chi Square quickly but also translates it into the probability that such a relationship could exist in our sample, were the null hypothesis (of no relationship in the population) true.

That is, it gives us the probability that we would commit a type I error if we were to accept the alternative hypothesis (rejecting a true null hypothesis) in a 'two tailed' test (i.e. where we made no prior assumptions about whether any difference in proportions meant that it were men or was women who were more likely to vote. This is referred to as 'significance testing'.

If we know how to interpret this, and use it alongside our other statistical tools, then using computer programmes, like dentistry with anaesthetics, is pure progress.

The bad side is that significance testing has spread like a disease, at times even to places that ought to be immune (like the editorial boards of respectable social science journals). Rather like the animals in Orwell's farm, where two legs were bad and four legs good, a tendency has grown up to slap a five per cent level significance test on everything and then to pay great attention to 'statistically significant' results and ignore the rest. You may even find published articles using significance tests to analyse data that is not based on random samples.

This is not so much ‘statistics’ as ‘carelessness’. It is important to keep in mind the logic of sampling inference. Sampling inference allows us to calculate precisely, using test statistics such as z scores or Chi Square, the probability that an (unknown) population parameter lies within a certain distance of our known sample statistic, or that an association which we find between two variables in our sample is one that also has a high probability of existing in the population. It follows that the width of our confidence intervals for means and proportions, and whether a particular result of a test for an association between two or more variables reaches the critical region of 5% or 1% significance, will depend, amongst other things, on the size of our sample.

With that warning, this is how you can ask our software to produce the Chi Square statistic for a contingency table, with the results for our table of vote_bin by gender reproduced in the table below.

 Source: ESS5 (UK) unweighted data

SPSS produces the ‘Pearson Chi-Square’ statistic that we calculated above by hand. It also produces a range of similar tests of significance. Rather than reporting simply whether or not the value for Chi Square falls within a particular critical region, SPSS reports the level of significance corresponding to the exact value of Chi Square.

Under the column headed 'Asymp Sig. (2 sided)' SPSS calculates the actual area under the tails of our distribution covered by values of Chi Square equal or greater to the value obtained. Thus it gives us the probability that we would obtain the result we have in our table, were our sample drawn from a population in which there was no association between our variables. It thus gives us the exact probability of committing a type I error if we reject our null hypothesis and accept the alternative: it is far too high in our case; we aim for a value below .05.

SPSS also reports how many cells have less than 5 observations expected in a cell of the table, because if there are more than 20% of cells with such small numbers, the value of Chi Sq becomes unreliable. It is usually possible to avoid this situation by recoding variables to collapse it into fewer categories.

Here, then, is the case that we do have a highly significant relationship between sex and smoking. Given ‘p’ = .006 then there is a less than 0.01 (1%) chance that there are actually proportionately as many or more male voters in our population than women voters. The pattern we have in our data would occur in less than 6 out of every thousand random samples of this size were there indeed no relationship in the population.

Findings like this often lead to sensationalist interpretations along the lines of 'women are the voters now' and so on, particularly in the popular press. But, using our more grounded and critical knowledge of quantitative analysis, even if we found there was a difference significant at the 1% level, what would it mean?

It would estimate our chances of committing a type 1 error if we reject the null hypothesis, given that we are drawing conclusions about a population from a sample. Because our sample is reasonably large, even quite modest differences in proportions or weak associations between variables show up. Even very weak correlations can be ‘statistically significant’.

Moreover, we have not yet considered any other potential relevant variables. Even if there is a significant correlation between two variables, other variables (e.g. age or social class indicators) could be equally or more important.