Conditional Probability and Bayes' Theorem

Conditional probability $P(A | B)$ is the probability of $A$ given that $B$ has occurred. Definition: $P(A | B) = P(A \cap B) / P(B)$, requiring $P(B) > 0$.

Independence

$A$ and $B$ are independent iff $P(A | B) = P(A)$ — knowing $B$ tells you nothing about $A$. Equivalently, $P(A \cap B) = P(A) P(B)$. Most "i.i.d." assumptions in statistics rest on independence; violating it (autocorrelation, network effects) breaks standard inference.

Bayes' theorem

Read: posterior = likelihood × prior / evidence. Bayes lets you flip the conditioning — from "probability of evidence given hypothesis" (often easy) to "probability of hypothesis given evidence" (what you want).

Classic example: medical test

A disease has prevalence $P(D) = 0.01$. A test has sensitivity $P(+ | D) = 0.99$ and specificity $P(- | \neg D) = 0.99$. If a random person tests positive, what's the probability they have the disease?

$P(D | +) = \frac{P(+ | D) P(D)}{P(+)} = \frac{0.99 \times 0.01}{0.99 \times 0.01 + 0.01 \times 0.99} = 0.5$

Despite a "99% accurate" test, half of positives are false alarms — because the disease is rare. Base rates matter.

Naive Bayes classifier

The simplest probabilistic classifier. Assume features are conditionally independent given the class: $P(x_1, \dots, x_n | C) = \prod_i P(x_i | C)$. Apply Bayes' rule: $P(C | x) \propto P(C) \prod_i P(x_i | C)$. Pick the class with the highest posterior. The independence assumption is rarely true but the classifier works surprisingly well — a good baseline.

Prior shapes inference

In Bayesian inference, the prior $P(H)$ encodes your beliefs before seeing data. As data accumulates, the posterior is dominated by the likelihood and the prior matters less. With small samples, the prior matters a lot — choosing it carefully (or analyzing sensitivity to it) is part of the modeling job.