p-values and Statistical Decisions — Section 8: Frequentist Inference

A $p$ -value answers a specific question: under the null hypothesis, how surprising is the data we observed (or something more extreme)?

p = P(\text{test statistic at least as extreme as observed} \mid H_0)

A small $p$ -value indicates the observed result would be rare if $H_0$ were true.

What $p$ -values are not:

$p$ is not $P(H_0 \mid \text{data})$ . Inverting the conditional requires Bayes and a prior.
$p < 0.05$ doesn't mean "true" — it means "significant at the $5\%$ level."
A non-significant result doesn't prove $H_0$ — absence of evidence is not evidence of absence.

Multiple testing is a real-world hazard. Run $20$ independent tests at $\alpha = 0.05$ and you expect $1$ false positive even when nothing is going on. Bonferroni controls family-wise error rate, Benjamini–Hochberg controls false discovery rate, and both matter when screening many factors, strategies, or signals.

A $p$ -value answers a specific question: under the null hypothesis, how surprising is the data we observed (or something more extreme)?

p = P(\text{test statistic at least as extreme as observed} \mid H_0)

A small $p$ -value indicates the observed result would be rare if $H_0$ were true.

What $p$ -values are not:

$p$ is not $P(H_0 \mid \text{data})$ . Inverting the conditional requires Bayes and a prior.
$p < 0.05$ doesn't mean "true" — it means "significant at the $5\%$ level."
A non-significant result doesn't prove $H_0$ — absence of evidence is not evidence of absence.