Title: Minimising prediction errors
Course: UGQuants (SSPS) > Section: Correlation and Regression

Minimising prediction errors

How do we know which line is the best line to summarise the relationship between two variables?  Sometimes we are lucky and get data from the real world that point to a very clear relationship between variables. Usually, however, real world data looks like this:

If our data look like this, how do we determine which line is best? The best line is the one that minimises errors.  In this case, errors are the gap between individual data points and the line.  We can identify these by measuring the space vertically and horizontally between any data point and the proposed line.

Our line allows us to predict values of y if we have data for x.  The relationship between our independent and dependent variables, between x and y, is summarised based on the data that we have.  This relationship, however, might not be true for every individual in our dataset.  Given someone's study habits we might predict that she will have a mark of 63 but she might have  actually earned a 54 or a 72.  We distinguish our predicted values from the real data that we have by using different names. We refer to the predicted values for the dependent variable as ŷ, or y hat, and the real values for the dependent variable, as y.  Error is the distance between what we have predicted (ŷ) and the real data (y), or expressed mathematically, y - ŷ.

IIn calculating the error of a particular regression line we have some below the line and some above . If we add all those distances together we will end up with zero.  For this reason, we calculate the distance between our data for the dependent variable (y) and the predicted value (ŷ), square them and then add them all up.  This gives us the sum of the squared errors (referred to as SSE).

Our best regression line is when the sum of the squared errors is minimised, or least. This is how we get the name Ordinary Least Squares regression, or OLS; it is when the squares of the errors are least.