Title: Logistic regression: modelling probabilities of a value rather than the value itself
Course: UGQuants (SSPS) > Section: Logistic Regression

Logistic regression: modelling probabilities of a value rather than the value itself

The European Social Survey in 2010 asked people if they voted in the last national election. The original variable derived from this question could take the following values:

1 Yes

2 No

3 Not eligible to vote

4 Refusal

5 Don’t know

6 No answer

To simplify the analysis, we create a purely binary indicator that takes the value 1 for everyone who answered ‘yes’, the value 0 for everyone who answered ‘no’, and a missing value for the rest, since we’re interested in the decision making process of why people choose to vote or not to vote.

The first problem we have to consider is that our dependent or Y variable takes only two values.  If we were to try to use simple linear regression, we would get the kind of result shown in the figure below.

The Y variable can only take two values, 0 or 1, so the results appear as two straight horizontal lines on our scatterplot.  There is a relationship to age (the mean age of those who vote is about 11.6 years older, at 52.13 compared to 40.74.  On our scatterplot this can be seen by the way the larger circles cluster towards different age points along the two lines (every circle stands for a number of cases – see the scale next to the scatterplot).  We can calculate a linear regression line, but it will not only be a poor fit, the residuals will vary systematically with the value of age, and our regression equation will happily predict impossible values above one and below zero, as well as values in between.

We can do better if, instead of estimating the value of Y directly, we try to estimate the probability that Y will take one value as opposed to the other for a given set of values of our independent variables.  Conventionally, we can label the two values of Y zero and one, and estimate the probability of Y taking the value one:  p(y=1).  When we estimate the probability of Y equalling one as exceeding 0.5, we can assign the value one; for less than 0.5 we can assign the value zero.

The trouble with probabilities, however, is that they are not distributed in a linear way and they are 'bounded', in the sense that they can range only from 0 to 1.  The same unit change in probability will have much more effect near values of 0.5 than at the extremes.

An example may help to explain this.

Suppose we were trying to relate ownership of a luxury car to income.  A change in annual income from £40,000 to £50,000 may make a lot of difference, but changes from £10,000 to £20,000, or £200,000 to £210,000 may have hardly any impact at all.  As we reach the extremes, the effect of the independent variables must decline because (unlike the case for an interval or ratio level dependent variable that can take any value) the values of our dependent variable have an upper and lower limit.  If we try to estimate probabilities directly, we will sometimes obtain results which exceed one or are negative, which makes no sense.

The problems worsen if we consider cases with multiple independent variables.  For example, imagine we found that 75% of men and 80% of women who had no post school qualifications had voted in the last national election, and that university education raised the number of voters for men by 30%.  If we were are asked to estimate the approval level for university educated women without any other information we would face a problem.  Whether we simply add 30% to 80% or multiply 80% by 1.3 we would still get results over 100%: a logical impossibility.

What do we do about this?