Title: Ordinary Least Squares
Course: UGQuants (SSPS) > Section: Multiple Regression

Ordinary Least Squares

Looking at the correlation between two variables can lead to poor conclusions if we are not careful.  For example, there might be a very strong correlation between the size of the stork population in a particular town and that town's human birth rate, but we might be foolish to think that such a correlation was evidence that storks do indeed deliver newborn children (although, who knows for sure?).  It would be better to look for a third variable (such as degree of urbanisation, levels of pollution, or perhaps even climate) that might determine both these variables, and thus reveal any correlation between them to be spurious, or to have a spurious component.

Multiple regression extends the regression equation between two variables, Ŷ = a + b(X) + ε,

by expressing Y (the dependent variable) as a function of two or more (up to any number) independent variables. It thus has a similar equation, where X_1 to X_n are the names of our independent variables and b_1 to b_n are the regression coefficients corresponding to them.

Ŷ = a + b_1X_1 + b_2X_2 + b_3X_3 + ... + b_nX_n + ε

This is called 'ordinary least squares' regression because the straight line is fitted to the points on our scatterplot by minimising the sum of the squared vertical distances between the data points and the line summarising them. We write Ŷ and not just Y, because all we can calculate is an estimate of Y.

It is 'linear' because we summarise our points with a straight line.  This means that we must first check that a straight, as opposed to a curved line, is indeed a good summary of our data.  We can do this by making scattergrams of the relationships between the dependent variable (Y) and each of the independent variables (X_i) we are interested in.  If necessary, we can transform one or more of our variables in order to make the relationship more or less linear.

Finally it is 'multiple' because we're dealing with many variables at once (at least three: one dependent and two or more independent variables).  Our interest is in the relationship between the dependent variable and each independent variable while controlling for one or more other independent variables.

Only a controlled experiment allows us to hold constant all the variables which might affect the relationship between our two variables of interest.  Given that experiments in the social sciences are rarely either practical or ethical, we have to employ statistical, rather than experimental, controls.  We have to try to analyse the patterns of relationships in our data to answer the following type of question: holding variables a, b, c and d constant, what is the relationship between x and y?  This is what multiple linear regression enables us to do in a practical way.