Title: Step 7: Checking for Multicollinearity
Course: UGQuants (SSPS) > Section: Multiple Regression

Step 7:   Checking for Multicollinearity

There is another important feature of regression that we should pay attention to and which serves as a reminder of the overriding importance of the theoretical models we use when working with more than one variable.  We can see the problem if we return to the WDI 2011 dataset and try regressing female life expectancy (using the untransformed LE_female variable) on the number of telephone lines per head of population.

Whether or not we transform this variable, it works very well indeed!  Even more startling, if we control for this variable and then add in lngdp as a second variable in our model, SPSS will tell us that lngdp only improves the model by 12.5%.  Does this mean that we have discovered a non-intuitive but 'scientific' explanation of life expectancy variation?  Should we argue that countries should install telephone lines to increase the life expectancy of their female citizens?

The answer is, of course, no.  There is a much simpler explanation for our pattern of results.  The number of telephone lines is highly correlated with GDP (r = 0.72).  Because of this, one variable tends to wipe out the contribution of the other when both are present in the model.  This problem is called multicollinearity.

It is useful to consider the original reason for carrying out multiple regression.  We want to find estimates of the association between an independent variable (X) and a dependent variable (Y), controlling for other relevant independent variables.  Imagine a situation in which one of the variables we wanted to control was perfectly correlated with X.  We would not in fact be able to control for it!  This is because there would be no variation in X left to account for once we had controlled for the other variable.

Think of the following example, using variables that take only two values.

 | Our dependent variable is whether or not a person passes an exam.
 | We have information on whether or not they revised for the exam and whether or not they attended all the lectures in the relevant course.

We are interested in looking at whether, controlling for lecture attendance, revision made a difference to the test outcomes.  Suppose we found that lecture attendance and revision were perfectly correlated (everyone who attended all the lectures did revision; everyone who did not revise also missed some lectures).  When we control for lecture attendance we can have only one value for revision: there is no variation left in it that we might associate with exam performance.  Try as we might, we would have no way that we could use our data to tell whether it was going to the lectures or studying that was important! (although we might find that they were both highly correlated with exam performance).

This is because we lack any examples of cases where a student missed some lectures but did revision, or went to all the lectures but did no revision.  We have no variation left in our independent variable once we control for the other independent.

Perfect correlation between independent variables is rare, but reasonably high correlation is quite common, especially when variables measure processes or countries that tend to go together.  In the above example economic progress tends to bring with it such things as more phone lines.  Where independent variables are highly correlated there is little variation left in one independent variable when controlling for each of the others in the model, and the estimates of the beta coefficients become more unstable: random variation in the data can produce wildly biased estimates of the contribution of each variable.  The overall value for R-squared will be unaffected, but the estimation of the individual beta coefficients can become unreliable.

This appears paradoxical.  Isn't it because of correlation between our independent variables that we want to do a multiple regression analysis in the first place? If there were no correlation between our independents then there would be no need to control ... The individual correlations between each dependent and our independent would tell us all we were trying to find out.  The paradox is resolved if we think again about what a regression model is trying to do.  It attempts to strip out the variation in our independent variable of interest (x) that is accounted for by its correlations with the other independents, so that an accurate picture of the direct association between x and y can be obtained.  If there is only moderate correlation between our independents, there is enough variation left in x once other independents have been controlled for to see clearly its association with y.  Because we have enough information, we can produce a good estimate of the likely value of the beta coefficient.  If, however, our independents are too highly correlated, one or more of them may have insufficient variation left to get a clear picture of this association.

There are two contrasting situations in which multicollinearity can occur.  It can be caused by having two or more variables in the model that are measuring much the same thing.  If this is the case the solution is simple.  We drop the redundant variable(s) and keep only those that are measuring different processes, or countries, etc.

The second situation occurs when highly correlated variables are measuring clearly distinct things, as in the example of the revision and lectures.  If these two things are highly correlated in the real world there is little we can do unless we can find new data in which for some reason the correlation has weakened.  However, the very existence of this high correlation may tell us something about the world: in this case that lecture attendance and revision go together.  What we cannot do, because of the nature of the world revealed in our evidence, is to discriminate between these two influences when examining their combined effect upon exam performance.  (It might be the case that what was important was revision, and lecture attendance did not matter at all, but until we find some observations of people who missed one or more lectures but did do the revision, we cannot possibly know this).
How do we know if there is a problem with multicollinearity?  We can easily check this by looking at the correlations between the independent variables.  As a rule of thumb, correlations above about 0.8 will probably cause some problems for the analysis.  If we have a lot of variables in the model, it could be a tedious process to check all the combinations.  The good thing is that we can get SPSS to do this for us, by choosing 'collinearity diagnostics' under the ‘statistics’ button in the regression dialog box.  This will produce a ‘variance inflation factor (VIF)’ and a 'tolerance' (These are the reciprocal of each other.) statistic adjacent to the beta coefficient for each variable.  The tolerance statistic is produced by subtracting the R-squared value obtained from regressing that variable on all the other independents from 1.  Thus, high values of tolerance (near 1) indicate that there is no collinearity problem, while values below about 0.4 suggest we need to be careful in our interpretation of the beta coefficient for that variable.  Multicollinearity only affects the beta coefficients for the variable(s) in the model that are highly correlated.  The estimates of the other beta coefficients in the model are not biased by it, nor is the estimate for the overall model R-squared affected.

If we produce a tolerance statistic for our lngdp and phone example it will have a value of 0.382: slightly below the suggested 0.4 threshold.  From our general knowledge we could hypothesise that it is unlikley that the number of phone lines 'cause' either economic progress or female life expectancy.  However, we would have a plausible theory, consistent with our evidence, were we to suggest that economic progress might affect both the number of phone lines and life expectancy.  We might, therefore, use the results of our model as evidence that the correlation between phone lines and life expectancy was either entirely or largely spurious.

We encountered what was actually a problem of multicollinearity earlier when we noted that the vast majority of low female life expectancy countries with high incidence of HIV were in Africa and poor.  In the absence of many poor non sub-Saharan African, high HIV incidence countries, it could be hard to determine what aspects of their experience are rooted in the social or political features of sub-Saharan Africa, or in the specific problems associated with AIDS.  Multicollinearity is more often a problem when dealing with aggregate data: data about organisations, areas, countries and so on, that result from measuring aggregates of individuals.  One potential reason for this is that such variables tend to have less variation than the individuals on which they are based.  One way in which this lower variation manifests itself is that variables based on such aggregates are more likely to show some association with each other.  If you explore the data in the WDI2011 dataset, you will soon find many variables that appear to be quite highly correlated.