Unit-2: Correlation and Linear Regression

1. Scatter Plots

A scatter plot is a graphical representation of the relationship between two variables. It displays data points for two numerical variables, with one variable on the x-axis and the other on the y-axis.

Purpose: To visualize the correlation or relationship between two variables.
Interpretation: - Positive correlation: Points trend upward. - Negative correlation: Points trend downward. - No correlation: Points are scattered randomly.

2. Correlation Coefficient

The correlation coefficient measures the strength and direction of the linear relationship between two variables. It ranges from -1 to 1.

r = 1: Perfect positive correlation.
r = -1: Perfect negative correlation.
r = 0: No correlation.

Properties of Correlation Coefficient

It is symmetric: $r_{x y} = r_{y x}$ .
It is unitless and scale-invariant.
It measures only linear relationships.
It is sensitive to outliers.

3. Karl Pearson’s Correlation Coefficient

Karl Pearson’s correlation coefficient ( $r$ ) is a measure of the linear correlation between two variables $X$ and $Y$ .

r = \frac{\sum (x_{i} - \bar{x}) (y_{i} - \bar{y})}{\sqrt{\sum {(x_{i} - \bar{x})}^{2} \sum {(y_{i} - \bar{y})}^{2}}}

r = \frac{Cov (X, Y)}{\sqrt{Var (X) \cdot Var (Y)}}

r = \frac{σ_{x y}}{σ_{x} \cdot σ_{y}}

Cov (X, Y) = E (X \cdot Y) - E (X) \cdot E (Y)

Var (X) = E (X^{2}) - {[E (X)]}^{2}

Var (Y) = E (Y^{2}) - {[E (Y)]}^{2}

E (X) = \frac{\sum X}{N}

E (X Y) = \frac{\sum X Y}{N}

E (X^{2}) = \frac{\sum X^{2}}{N}

Cov (X, Y) = E [(X - μ_{x}) (Y - μ_{y})] = \frac{\sum (X - \bar{X}) (Y - \bar{Y})}{N}

Where:

$\bar{x}$ and $\bar{y}$ are the means of $X$ and $Y$ , respectively.
$x_{i}$ and $y_{i}$ are individual data points.

4. Spearman’s Rank Correlation Coefficient

Spearman’s rank correlation coefficient $r_{s}$ measures the strength and direction of the monotonic relationship between two variables. It is based on the ranks of the data.

Between two variables when there is no tie during rank:

r_{s} = 1 - \frac{6 \sum d_{i}^{2}}{n (n^{2} - 1)}

Between two variables when there is a tie during rank:

r_{s} = 1 - \frac{6 (\sum d_{i}^{2} + \frac{\sum m (m^{2} - 1)}{12})}{n (n^{2} - 1)}

Correction function:

\frac{\sum m (m^{2} - 1)}{12}

Where:

$d_{i}$ is the difference between the ranks of corresponding values of $X$ and $Y$ .
$n$ is the number of observations.
$m$ is the multiplication of repeated rank.

5. Linear Regression

Linear regression is a statistical method used to model the relationship between a dependent variable $Y$ and one or more independent variables $X$ .

For $Y$ on $X$ :

Y = a + b X

(y - y^{'}) = (r \cdot \frac{σ_{y}}{σ_{x}}) (x - \bar{x})

b_{y x} = (r \cdot \frac{σ_{y}}{σ_{x}}) = \frac{Cov (X, Y)}{Var (X)}

For $X$ on $Y$ :

X = a + b Y

(x - \bar{x}) = (r \cdot \frac{σ_{x}}{σ_{y}}) (y - \bar{y})

b_{x y} = (r \cdot \frac{σ_{x}}{σ_{y}}) = \frac{Cov (X, Y)}{Var (Y)}

Where:

$Y$ : Dependent variable.
$X$ : Independent variable.
$a$ : Y-intercept.
$b$ : Slope of the line.
$b_{y x}$ : Regression coefficient of $Y$ on $X$ .
$b_{x y}$ : Regression coefficient of $X$ on $Y$ .

Properties of Linear Regression

The point $\bar{x}, \bar{y}$ is the intersection point of the two regression lines.
For perfect correlation, both regression lines coincide.
The correlation coefficient is the geometric mean of the regression coefficients: $r = \pm \sqrt{b_{y x} \cdot b_{x y}}$
If one regression coefficient is greater than unity, the other must be less than unity.
Regression coefficients are independent of changes in origin but not of scale: $b_{a y + c . d x + e} = (\frac{a}{b}) b_{y x}$
The modulus value of the arithmetic mean of the regression coefficients is not less than the modulus of the correlation coefficient ( $r$ ): $\frac{b_{y x} \cdot b_{x y}}{2} \geq \sqrt{b_{y x} \cdot b_{x y}}$
The regression line minimizes the sum of squared errors (SSE).
The slope $b$ represents the change in $Y$ for a unit change in $X$ .
The intercept $a$ represents the value of $Y$ when $X = 0$ .
The regression line passes through the point $\bar{x}, \bar{y}$ .

Angle Between Two Lines of Regression

\tan θ = \frac{(1 - r^{2})}{| r |} \cdot \frac{σ_{x} σ_{y}}{σ_{x}^{2} + σ_{y}^{2}}

For perfect correlation:

r = \pm 1

\tan θ = 0 (θ = 0 °)

This means the lines coincide.

For no correlation:

r = 0

\tan θ = \infty (θ = 90 °)

This means the equations of the lines are:

y = \bar{y} (for Y on X)

X = \bar{X} (for X on Y)

If the lines are given:

\tan θ = | \frac{m_{1} - m_{2}}{1 + m_{1} m_{2}} |

m_{1} = b_{y x}

m_{2} = \frac{1}{b_{x y}}

The ratio of variances is the same as the ratio of regression coefficients:

\frac{σ_{y}^{2}}{σ_{x}^{2}} = \frac{b_{y x}}{b_{x y}}

Formulas for Linear Regression

Slope $b$ :

b = \frac{\sum (x_{i} - \bar{x}) (y_{i} - \bar{y})}{\sum {(x_{i} - \bar{x})}^{2}}

Intercept $a$ :

a = \bar{y} - b \bar{x}

MCQ Questions

1. What does a correlation coefficient of 0 indicate?

a) Perfect positive correlation
b) Perfect negative correlation
c) No correlation
d) None of the above

Answer: c) No correlation

2. In Spearman’s rank correlation, what does $d_{i}$ represent?

a) Difference between ranks of corresponding values
b) Sum of ranks of corresponding values
c) Product of ranks of corresponding values
d) None of the above

Answer: a) Difference between ranks of corresponding values