Title: Step 8: Checking the model
Course: UGQuants (SSPS) > Section: Multiple Regression

Step 8:   Checking the model

Residuals

When we construct a model using regression, we are trying to 'fit' it as best we can to the data.  If we had a perfect model (something which never happens: life is rarely pure and never simple) there would be no differences between the values of Y predicted by our model on the basis of the values of our independent X variables, and the observed values of Y recorded in our dataset for every case. Conversely, if even the best model we can make has a rather poor fit, these differences will be quite large.  The difference between the observed value for Y and the value for it predicted by our model is known as the residual.  Recall that our regression model is basically an attempt to minimise these residuals (defined as the sum of their squares).  The nature and pattern of our residuals, therefore, tells us a great deal about our model.

Other things equal, we would like:

(a) | Residuals with no systematic pattern, so that error is randomly distributed and thus approximates a normal curve.
(b) | Residuals with a mean close to zero; our predicted value of Y, on average, should be neither an overestimate nor an underestimate.
(c) | Small residuals; that is to say, we want our model to produce accurate predictions, or account for a large proportion of the variation in our dependent variable.  What we do not want are too many outliers and certainly not cases which distort our model.
(d) | Residuals that are uncorrelated with the values of the dependent variable Y (known as ‘homoscedasticity’).

Generally, the more our residuals conform to these qualities, the better our model.  When we have outliers in our predictions, a case or cases that fit very badly, then this will affect all these desirable qualities of residual patterns.  We have already seen that one way we can examine the quality of the fit is to look for cases that fit very badly:  When we were building a model for the relationship between lngdp, lnHIV and LE_female in Sub-Saharan countries we found that Cape Verde and Mauritius were examples of just such individual outliers.  For their quite high female life expectancies, they had an unexpectedly high HIV prevalence.  Two cases (plus the incorrect value for Namibia) pulled the regression line in a different direction to where it would otherwise have been.  We could simply see by visual inspection of our scatterplot that each of them was an 'outlier': a case much further from the position predicted by the regression line than other cases in our dataset.  Visual inspection is not always enough though, so SPSS enables us to obtain a good deal of useful information about our residuals whenever we run a model to make a more informed judgement.

Visual examination

When SPSS carries out a linear regression we can also get it to calculate a range of new variables for the dataset that record such things as the value of the residual (standardised or unstandardised), or the value predicted in the regression model for each case.  We can then use these to produce scatterplots of the residuals aginst the values of the independent variables, or the predicted values of the dependent variable to look for patterns in the residuals.  We can also get SPSS to produce what is called a normal probability plot.  This plot basically shows the observed cumulative distribution of the values of the standardised residuals (on the X axis) against their expected normal distribution (on the Y axis).  If the residuals are normally distributed the result will be a straight line.  If they are not normally distributed, this will show up as deviations from this line.  As the following diagram shows, the normal probability plot for the regression of LE_female on lngdp for low female life expectancy countries is not very straight.

However, when we include the lnHIV and Reg4 variables in our model, things improve considerably, as the following diagram shows.

When you obtain residuals that are not normally distributed, as indicated in a normal probability plot, it is time to use the capability of SPSS to create variables that store the residual values for each case in order to examine the pattern of residuals in greater depth.  Once we know where our model does well or badly we will usually have some useful clues about how to go about improving it, just as we did when we asked ourselves why per capita GDP did so poorly in predicting female life expectancy in poor countries concentrated in Africa.

Animation on Producing Plots and Saving Residuals

We can undertake the same analysis by asking for a histogram of the standardised residuals instead.  This will produce a histogram of the value of the standardised residuals for all observations.  We need to check that it approximates to a normal curve.  You can get SPSS to store the values of the residuals and the predicted values of Y (referred to as Zresid and Zpred) and to produce a scatterplot of these two variables to test for heteroscedasticity.

We can also use both the standardised and unstandardised residual values to check for outliers.  As a rule of thumb, a standardised residual with a value of more than three standard deviation units (+/ - 3) is worth looking at.  We would need to check that there is not a data entry error before considering if there are good reasons why this case may be atypical.

Cook's distance statistic

Previously we identified outliers by visual inspection and examined their influence by re-running the model with these cases excluded.  However, rather than relying on the often subjective interpretation of our eyes, we can calculate a value for the influence of each case on the model.  This value can be measured by a statistic known as 'Cook's distance'.  SPSS can calculate this for any regression we run and create a new variable in our dataset with its value for each case.  We can either inspect this directly (if we have a manageable number of cases), or, if we have a large number of cases, we can plot a histogram, or get SPSS to calculate descriptive statistics just as we would for any other variable.  As a rule of thumb, it is worth investigating any cases where Cook's D is greater than 4/n (4 divided by the number of our observations).

If we try doing this for the regression of LE_female on lnHIV for our Sub-Saharan countries, we will see that the values for Cook's distance for our outliers are higher than 4/45 = 0.089 (Cape Verde: 0.17350; Mauritius: 0.15726; Namibia: 0.1111) and that they are indeed the highest amongst all of the countries’ values.

Animation of Cook’s Distance