Logistic Regression

Logistic regression models the probability of a binary outcome as a logistic function of a linear combination of inputs. Despite the name, it's a classification model — the most-used baseline for tabular classification problems.

The model

where $\sigma(z) = 1 / (1 + e^{-z})$ is the logistic sigmoid. The linear combination is squashed into $(0, 1)$ to be a valid probability.

Log-odds

The model is linear in the log-odds: $\log(p / (1 - p)) = \beta_0 + \sum \beta_j x_j$. Each coefficient $\beta_j$ is the change in log-odds per unit change in $x_j$. Exponentiating gives an odds ratio: $e^{\beta_j}$ is the multiplicative change in odds per unit of $x_j$.

Estimation

Maximize the log-likelihood: $\sum_i [y_i \log p_i + (1 - y_i) \log (1 - p_i)]$. No closed form; solved iteratively (Newton-Raphson, gradient descent). Convex in $\beta$, so the global optimum is unique.

Decision threshold

For a binary decision, classify $\hat{y} = 1$ if $p > 0.5$ (or any other threshold). Different thresholds trade off precision and recall — for imbalanced classes or asymmetric costs, the optimal threshold isn't 0.5.

Multinomial logistic

Extension to $K$ classes. One linear combination per class, softmaxed: $P(y = k | x) = e^{\beta_k^T x} / \sum_j e^{\beta_j^T x}$. Equivalent to "softmax regression" or the last layer of a neural network.

Why logistic, not linear

You could fit a linear model directly to 0/1 labels (LPM). It gives unbiased coefficients but predicted probabilities can exceed $[0, 1]$, and the constant-variance assumption is wildly violated. Logistic is the right model when the outcome is genuinely binary.