Title: Type I + II errors and significance
Course: UGQuants (SSPS) > Section: Inference

Type I + II errors and significance

Because there is a chance of error, we can never be absolutely sure of our findings: we can never claim to have 'proven' something or to have found 'the truth'.

When we examine associations between two (or more) variables, we therefore test hypotheses. There are two types of hypotheses:

- the null hypothesis: this hypothesis claims that there is no association between the variables. That means, a change in one variable does not lead to a change in the other variable. For example, your age has no effect on your gender.
- the alternative hypothesis: this hypothesis states that there is an association between the variables.

When writing up your research report, you will present the alternative hypothesis as your hypothesis, but you will seek to reject the null hypothesis. This is caused by the idea of falsifiability as it was formulated by Karl Popper: in science, we can never prove something right with 100% certainty, but we can try to find a case in which the statement we make is wrong. So, we test the null hypothesis and when we find no empirical evidence supporting it, we can provisionally reject the null hypothesis and accept the alternative hypothesis.

Type I and type II errors

When we decide on the size of the chance we are willing to take of generalising a result from our sample that is actually incorrect for our complete population of interest, we are determining the chance of making a type I error.

A type I error is committed when the null hypothesis was rejected although in fact it applied. That means, we conclude that there is an association between the variables when in truth there is none.

A type II error is the opposite: we assume there is no association when in truth there is one.

This chance is often referred to as significance (short Sig. or p) and you will encounter it throughout this course. For most measures of association and coefficients in regression analysis, you can produce a significance value telling you how big your chance of committing a type I error is. This value is displayed as number between 0 and 1, and your aim is for it to be as small as possible (usually below 0.05 - that is, below the 5% threshold mentioned earlier).

Keep in mind: significance tells you nothing about the importance of your findings. It merely tells you how likely it is that your findings would be found if you had enough research funding to examine every case in your population instead of just a sample.