Law of Large Numbers — Section 7: Limit Theorems

The Law of Large Numbers (LLN) is the mathematical justification for "more data is better." For independent identically distributed random variables $X_1, X_2, \dots$ with finite mean $\mu$ , the sample average converges to $\mu$ :

\bar{X}_n = \frac{1}{n}\sum_{i=1}^{n} X_i \to \mu \quad \text{as } n \to \infty

There are two flavors. The weak LLN says convergence holds in probability — for any $\epsilon > 0$ , $P(|\bar{X}_n - \mu| > \epsilon) \to 0$ . The strong LLN says it holds almost surely — the random sequence converges with probability $1$ . The strong version is genuinely stronger; the weak follows from it.

LLN is what makes Monte Carlo simulation work, what justifies A/B testing, and what underpins empirical risk estimation. Average enough samples and you get the expectation, with the precision improving like $1/\sqrt{n}$ (per the CLT).

Watch out: LLN requires finite mean. For Cauchy and other distributions where $E[X]$ doesn't exist, sample averages never stabilize — they keep jumping by orders of magnitude as $n$ grows.

\bar{X}_n = \frac{1}{n}\sum_{i=1}^{n} X_i \to \mu \quad \text{as } n \to \infty

Watch out: LLN requires finite mean. For Cauchy and other distributions where $E[X]$ doesn't exist, sample averages never stabilize — they keep jumping by orders of magnitude as $n$ grows.