Title: From correlation to regression
Course: UGQuants (SSPS) > Section: Correlation and Regression

From correlation to regression

Although the Pearson’s product moment correlation coefficient (i.e. Pearson’s r) provides us with a useful way to summarise the relationship between two variables, there are different ways to do this.

Let’s assume, for the moment, that we want to predict the value of something, like a student’s mark in a course. Our best bet is usually to guess the mean. If we know that the typical student at a university receives a 55 in a particular course, and we were asked to guess a particular student’s mark, our best guess would be that she received a 55.  We might be wrong a lot of the time, but we’d probably be close to the right answer.

What if we have a little bit of information about the student. Let’s say, for example, that we know that as the number of hours spend studying for exams increases, educational attainment increases.  Perhaps we know that if a student studies 200 hours per year the average student mark is 55, but if a student studies 300 hours average is 59 and if they study 400 hours the average is 63.  In this imaginary world, each time the number of hours spent studying increases by 100, a student’s average mark increases by four points.

If we were to meet a student on the street and wanted to guess her grade we could inquire about the number of hours she spent studying for her exams.  Let us say that she spent 350 hours studying: we would guess that she received a 61.  If we know that a 100 unit increase in hours produces a four point increase in marks, it is logical to assume that a 50 unit increase in hours would produce a two point increase in marks.  This is easier to see if we graph this where the x axis refers to hours spent studying and the y axis refers to marks.

It is relatively easy for us to identify possible marks if we are dealing with regular increases in hours, but what if the number of hours spent studying was 375.6 or 479?

If we draw a line to summarise the relationship between our two variables, between study time and average marks, we can use the line to determine predicted marks for any level of study time.

This, in essence, is regression. We take information we have about the relationship between variables, summarise that relationship and then make predictions about dependent variables (such as marks) whenever we have information about independent variables (such as hours spent studying).  With correlation we are concerned with the strength of association between two variables. In regression we are also concerned with identifying one variable as dependent on the other(s) and specifying the nature of the relationship.

When we summarise the relationship between variables in this way we use a line that represents the most typical results.  The equation for any straight line is that which we might remember from secondary school: y=a + bx

The equation for a regression line based on two variables (x and y) is very similar. It is y = a + bx + e

Here, ‘b’ is the slope of the line, sometimes referred to as rise over run (or change in y over the change in x, or the height of the line divided by the width of the line).  ‘a’ is the starting point, the base line amount, or constant.  We sometimes refer to this number as the intercept, for it gives us the height of the line when it crosses the y (vertical) axis.  ‘e’ is the error term.  In our earlier example of hours spent studying and marks we found a very easy relationship to examine, but with real world data the relationship is not nearly so neat and, since we’re usually looking at data from a sample rather than the entire population, we have to assume that we haven’t captured all of the possible variation that exists in the real world.  We include an error term in the equation to remind us of this.

Returning to our earlier ‘ideal world’ situation, we can calculate the slope of the line.  The rise (or change in y, or height of the line) is calculated by taking the highest and lowest scores for the line on the y axis, in this case 63 and 55.  The run (or change in x, or length of the line) is calculated by taking the highest and lowest scores for the line on the x axis, in this case 400 and 200.

b=.04

When we are dealing with data we have collected, however, we often don’t have a few data points that fit neatly onto a single line.  Often we have something that looks like this:

When we want to identify the slope of the regression line that best summarises the relationship between our independent and dependent variables, we use a different formula.  This formula takes account of the variation that exists in our data and acknowledges that data points do not always lie exactly on the line.  Each data point has of course two values, one for the independent variable (x) and one for the dependent variable (y).  We can summarise the position of each datapoint as (x, y).  A data point with coordinates (5, 8) has a score of 5 for the independent variable and a score of 8 for the dependent variable.  To calculate our regression line slope, or regression coefficient, we have to identify

 | The score of each datapoint for the independent variable (x)
 | The score of each datapoint for the dependent variable (y)

Taking into account all the data in our sample we have to calculate:

 | The mean for the independent variable
 | The mean for the dependent variable

We now have everything we need to calculate the regression coefficient.  For each datapoint we identify the distance between its score for the independent variable and the mean for the independent variable (or x-).  We then calculate the distance between the score for the dependent variable and the mean for the dependent variable (or y-).  We multiple these by each other to get a sense of how far our case is from the mean of both the variables of interest.  We do this for each case in our dataset and then add these all up.  We divide this figure by the variance for the independent variable.  The resulting formula is this:

The result tells us the amount that the dependent variable (or y) changes, either up or down, for each one unit increase in x.  Put another way, b is the expected change in y given a unit increase in x.  If our b was 5.62, we would say a one unit increase in the independent variable produces a 5.62 increase in the dependent variable.

This video gives you an overview of regression analysis: