Correlation and Covariance

Covariance measures how two variables move together. Correlation rescales it to $[-1, 1]$ for comparability across units.

Covariance

$\text{Cov}(X, Y) = E[(X - \mu_X)(Y - \mu_Y)]$. Positive when $X$ and $Y$ tend to be above (or below) their means together. Negative when one is above and the other below. Sign matters; magnitude is in units of $X \cdot Y$ — hard to interpret.

Pearson correlation

$\rho = \text{Cov}(X, Y) / (\sigma_X \sigma_Y)$. Dimensionless, in $[-1, 1]$. $\rho = 1$ means perfect positive linear relationship, $\rho = -1$ perfect negative, $\rho = 0$ no LINEAR relationship — but $X$ and $Y$ could still be perfectly dependent in a nonlinear way (think $Y = X^2$ on symmetric data: $\rho = 0$).

Spearman correlation

Pearson applied to the ranks rather than the values. Measures monotonic (not necessarily linear) relationships. Robust to outliers. Use when the relationship is monotonic but possibly curved.

Correlation is not causation

The most-quoted mantra in statistics. High correlation could mean: $X$ causes $Y$, $Y$ causes $X$, both are caused by a third variable $Z$, or pure coincidence in a small sample. Causal claims require either an intervention (randomized experiment) or strong identifying assumptions (instrumental variables, regression discontinuity, difference-in-differences).

Multicollinearity

When predictors in a regression are highly correlated with each other, the model's coefficients become unstable — small changes in data shift them wildly. Detect with variance inflation factors (VIF); fix by dropping variables, combining them (PCA), or regularization.

Partial correlation

The correlation between $X$ and $Y$ controlling for one or more other variables. Useful for asking "is the apparent relationship between $X$ and $Y$ explained by $Z$?"