Central Limit Theorem

The Central Limit Theorem is one of the most consequential results in probability — it explains why the Normal distribution shows up everywhere.

For independent identically distributed random variables $X_1, X_2, \dots$ with mean $\mu$ and finite variance $\sigma^2$, the standardized sample average converges in distribution to a standard Normal:

In other words, once $n$ is reasonably large, the distribution of the sample mean looks Gaussian — regardless of the original distribution. Roll a die a hundred times and average; the average's distribution is approximately Normal even though a single roll is uniform.

There are caveats. CLT requires finite variance. It fails for Cauchy, and fails for power-law tails with index $\alpha \le 2$. Convergence rate depends on the skewness and kurtosis of the underlying distribution; for highly skewed cases, $n$ in the hundreds may not be enough.

CLT is why so many estimators are approximately Normal, and why Normal-theory confidence intervals work even when the underlying data isn't Normal. It's the engine behind almost all of frequentist statistics.