Lecture #24: A History of Correlation

Example #1: Violent Crime Rate in 1990 vs. 2000

Understanding the persistence of violent crime in cities can help determine which cities are being successful in reducing it and which cities are not. To start the research, we will need to determine whether there is a relationship between the violent crime rate in 1900 and in 2000.

Note: At this point, we only want to know whether there is a significant linear relationship. How could we determine this?

Example #1: Violent Crime Rate in 1990 vs. 2000

Questions:

From this graphic, does there appear to be a correlation between these two variables?

What is it about the graphic makes you think there is a relationship?

The Theory

That is the key question:

What is it about the graphic tells us there is a relationship between these two variables?

This was the question that led us – eventually – to our modern measure of correlation.

The next several slides walk through the history of many answers to this question.

Concordance

The first attempt compared the number of concordant points to discordant points.

A concordant point is one that supports a positive relationship between the two variables.

A discordant point is one that supports a negative relationship between the two variables.

Example #1: Violent Crime Rate in 1990 vs. 2000

The original correlation measure was the number of concordant points minus the number of discordant points, divided by the sample size.

This “concordance ratio” measure for this data is \(\frac{(47-4)}{51} = 0.84\).

Concordance Ratio

This measure has the strengths:

• It is easily calculated.
• The measure ranges between \(-1\) and \(+1\).

This measure had the weaknesses:

• This measure ignores information about the points (specifically, how far from the center).

Covariance

Covariance, \(cov(x, y)\) or \(s_{xy}\), is a measure of how much the two variables vary together, taking into consideration the data values.

\[ s_{xy} = \frac{1}{n-1} \sum_{i=1}^n (x_i - \overline{x}) (y_i - \overline{y}) \]

Check that this measure takes into consideration both concordance and the distance to the center.

Covariance

The covariance is cov(vcrime90, vcrime00) \(= 88047.41\).

Covariance

This measure has the strengths:

• It is based on concordance.
• It is based on how far the point is from the center.

This measure had the weaknesses:

• This measure does not range between \(-1\) and \(+1\), and, as such, cannot be compared across data sets.

To fix this weakness, the “correlation” was devised.

Correlation

Correlation is a standardized covariance (Bravais, 1844).

Check that the following measure takes into consideration these two things:

1. the concordance
2. the distance to the center

all the while ensuring that it ranges between \(-1\) and \(+1\):

Correlation

\[ cor(x,y) = r_{xy} = \frac{s_{xy}}{s_x s_y} = \frac{\sum (x_i - \overline{x}) (y_i - \overline{y})}{\sqrt{\sum (x_i - \overline{x})^2} \sqrt{\sum (y_i - \overline{y})^2}} \]

This is the formula for the “sample correlation”. It is a good estimator of the population correlation \(\rho_{xy}\), which is what we like to understand.

Correlation

The correlation is cor(vcrime90, vcrime00) \(= 0.936469\).

Correlation

Since correlation ranges between \(-1\) and \(+1\), we can compare correlation between different pairs of variables.

Correlation Test

Rahman (1944) showed, for correlations near 0, that

\[ r \sim Normal \left( \rho, \frac{1 - \rho^2}{n-2} \right) \]

Correlation Test

Confidence intervals when \(r\) is far from 0 are a bit trickier, however.

In R, the command to perform the correlation test is cor.test(x, y).

Example #2: Property Crime Rate in 1990 vs. 2000

Are the property crime rate in 1900 and in 2000 significantly correlated?

Here is the code to answer this question:

dt = read.csv("http://rfs.kvasaheim.com/data/crime.csv")
attach(dt)

cor.test(pcrime90, pcrime00)

    Pearson's product-moment correlation

data:  pcrime90 and pcrime00
t = 9.3781, df = 49, p-value = 1.623e-12
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.6748213 0.8821406
sample estimates:
      cor 
0.8013737 

Example #2: Property Crime Rate in 1990 vs. 2000

Brief Conclusion:

Because the \(p\)-value is much less than our \(\alpha = 0.05\), we reject the null hypothesis that there is no correlation between the property crime rates in 1990 and in 2000. In fact, we are 95% confident that the true correlation is between 0.67 and 0.88. The graphic on the next page illustrates this.

Example #2: Property Crime Rate in 1990 vs. 2000

Example #3: Wealth and Professionalism

Are the average wealth in the state and the level of professionalism in its legislature correlated?

Here is the code to answer this question:

cor.test(gspcap00, profleg)

    Pearson's product-moment correlation

data:  gspcap00 and profleg
t = 2.3585, df = 48, p-value = 0.02247
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.04823515 0.55116484
sample estimates:
      cor 
0.3222562 

Example #3: Wealth and Professionalism

Brief Conclusion:

Because the \(p\)-value of 0.0225 is less than our \(\alpha = 0.05\), we reject the null hypothesis that there is no correlation between the average wealth in the state and the level of professionalism in its legislature. In face, we are 95% confident that the true correlation is between 0.05 and 0.55. The graphic on the next page illustrates this.

Example #3: Wealth and Professionalism

Example #4: Wealth and Education

Are the average wealth in the state and the school enrollment correlated?

Here is the code to answer this question:

cor.test(gspcap00, enroll00)

    Pearson's product-moment correlation

data:  gspcap00 and enroll00
t = 1.767, df = 49, p-value = 0.08346
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.03306579  0.48745373
sample estimates:
      cor 
0.2447481 

Example #4: Wealth and Education

Brief Conclusion:

Because the \(p\)-value of 0.0835 is greater than our \(\alpha = 0.05\), we should not reject the null hypothesis. We did not detect a relationship between the average wealth in the state and the enrollment percentage. In fact, we are 95% confident that the true correlation is between \(-0.03\) and \(0.49\). The graphic on the next page illustrates this.

Example #4: Wealth and Education

Example #5: Wealth and Crime

Are the average wealth in the state and the 2000 violent crime rate correlated?

Here is the code to answer this question:

cor.test(gspcap00, vcrime00)

    Pearson's product-moment correlation

data:  gspcap00 and vcrime00
t = 3.5628, df = 49, p-value = 0.0008289
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.2034498 0.6481614
sample estimates:
      cor 
0.4536012 

Example #5: Wealth and Crime

Brief Conclusion:

Because the \(p\)-value of 0.0008 is less than our \(\alpha = 0.05\), we should reject the null hypothesis. We did detect a relationship between the average wealth in the state and the violent crime rate. In fact, we are 95% confident that the true correlation is between 0.20 and 0.65.

Example #5: Wealth and Crime